Skip to contentSkip to conversation

New: KVM guests randomly get I/O errors on VirtIO based devices

Bug 110441 [email protected] 1 participant 4 messages Jan 6, 2016 – Oct 15, 2021 KVM

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=110441

            Bug ID: 110441
           Summary: KVM guests randomly get I/O errors on VirtIO based
                    devices
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.16.7-ckt11-1+deb8u5
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

We've been seeing a strange bug in KVM guests hosted by a Debian jessie box (running 3.16.7-ckt11-1+deb8u5 on x86-64),

Basically, we are getting random VirtIO errors inside our guests, resulting in stuff like this

[4735406.568235] blk_update_request: I/O error, dev vda, sector 142339584
[4735406.572008] EXT4-fs warning (device dm-0): ext4_end_bio:317: I/O error -5
writing to inode 1184437 (offset 0 size 208896 starting block 17729472)
[4735406.572008] Buffer I/O error on device dm-0, logical block 17729472
[ ... ]
[4735406.572008] Buffer I/O error on device dm-0, logical block 17729481
[4735406.643486] blk_update_request: I/O error, dev vda, sector 142356480
[ ... ]
[4735406.748456] blk_update_request: I/O error, dev vda, sector 38587480
[4735411.020309] Buffer I/O error on dev dm-0, logical block 12640808, lost
sync page write
[4735411.055184] Aborting journal on device dm-0-8.
[4735411.056148] Buffer I/O error on dev dm-0, logical block 12615680, lost
sync page write
[4735411.057626] JBD2: Error -5 detected when updating journal superblock for
dm-0-8.
[4735411.057936] Buffer I/O error on dev dm-0, logical block 0, lost sync page
write
[4735411.057946] EXT4-fs error (device dm-0): ext4_journal_check_start:56:
Detected aborted journal
[4735411.057948] EXT4-fs (dm-0): Remounting filesystem read-only
[4735411.057949] EXT4-fs (dm-0): previous I/O error to superblock detected

(From an Ubuntu 15.04 guest, EXT4 on LVM2)

Or,

Jan 06 03:39:11 titanium kernel: end_request: I/O error, dev vda, sector
1592467904
Jan 06 03:39:11 titanium kernel: EXT4-fs warning (device vda3):
ext4_end_bio:317: I/O error -5 writing to inode 31169653 (offset 0 size 0
starting block 199058492)
Jan 06 03:39:11 titanium kernel: Buffer I/O error on device vda3, logical block
198899256
[...]
Jan 06 03:39:12 titanium kernel: Aborting journal on device vda3-8.
Jan 06 03:39:12 titanium kernel: Buffer I/O error on device vda3, logical block

99647488

(From a Debian jessie guest, EXT4 directly on a VirtIO-based block device)

When this happens, it affects multiple guests on the hosts at the same time. Normally they are severe enough that they end up with a r/o file system, but we've seen a few hosts survive with a non-fatal I/O error. The host's dmesg has nothing interesting to see.

We've seen this happen with quite heterogeneous guests:

Debian 6, 7 and 8 (Debian kernels 2.6.32, 3.2 and 3.16) Ubuntu 14.09 and 15.04 (Ubuntu kernels)
32 bit and 64 bit installs.

In short, we haven't seen a clear characteristic in any guest, other than the affected hosts being the ones with some sustained I/O load (build machines, cgit servers, PostgreSQL RDBMs...). Most of the times, hosts that just sit there doing nothing with their disks are not affected.

The host is a stock Debian jessie install that manages libvirt-based QEMU guests. All the guests have their block devices using virtio drivers, some of them on spinning media based on LSI RAID (was a 3ware card before, got replaced as we were very suspicious about it, but are getting the same results), and some of them based on PCIe SSD storage. We have some other 3 hosts, similar setup except they run Debian wheezy (and honestly we're not too keen on upgrading them yet, just in case), none of them has ever shown this kind of problem

We've been seeing this since last summer, and haven't found a pattern that tells us where these I/O error bugs are coming from. Google isn't revealing other people with a similar problem, and we're finding that quite surprising as our setup is quite basic.

This has also been reported downstream at the Debian BTS as Bug#810121
(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=810121).

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw

[Bug 110441] KVM guests randomly get I/O errors on VirtIO based devices

https://bugzilla.kernel.org/show_bug.cgi?id=110441

Jordi Mallach <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #1 from Jordi Mallach <[email protected]> --- Sorry for the noise. This is actually caused by os-prober opening the devices and causing corruption and mayhem. See
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788062 for details.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw

[Bug 110441] KVM guests randomly get I/O errors on VirtIO based devices

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=110441

Jerick Fischer ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #2 from Jerick Fischer ([email protected]) --- (In reply to Jordi Mallach from comment #1)

Quoted 3 lines

Sorry for the noise. This is actually caused by os-prober opening the devices and causing corruption and mayhem. See
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788062 for details.

Thank you much for sharing your knowledge and a reference, much appreciated.

Respectfully,
RJ from https://www.sanjosetruckingcompany.com/

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 110441] KVM guests randomly get I/O errors on VirtIO based devices

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=110441

Ahmed Sayeed ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from Ahmed Sayeed ([email protected]) ---
ake: Entering directory '/home/Christian/binutils-gdb/cygwin-obj/gdb'
CXXLD gdb.exe http://www.compilatori.com/computers/smartphones/
cp-support.o: in function `gdb_demangle(char const*, int)':
http://www.acpirateradio.co.uk/services/ios15/
/home/Christian/binutils-gdb/cygwin-obj/gdb/../../gdb/cp-support.c:1619:(.text+0x5502):
http://www.logoarts.co.uk/property/lidar-sensor/ relocation truncated to fit:
R_X86_64_PC32 against undefined symbol
http://www.slipstone.co.uk/property/hp-of-cars/ `TLS init function for thread_local_segv_handler' /home/Christian/binutils-gdb/cygwin-obj/gdb/../../gdb/cp-support.c:1619:(.text+0x551b):
http://embermanchester.uk/property/chat-themes/ relocation truncated to fit:
R_X86_64_PC32 against undefined symbol `TLS init function for thread_local_segv_handler'
collect2: error: ld returned 1 exit status
http://connstr.net/property/mars-researches/
make: *** [Makefile:1881: gdb.exe] Error 1
make: Leaving directory '/home/Christian/binutils-gdb/cygwin-obj/gdb'
http://joerg.li/services/kia-rio-price/

$ g++ -v
Using built-in specs. http://www.jopspeech.com/technology/thunderbolt-4/
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/10/lto-wrapper.exe
Target: x86_64-pc-cygwin http://www.wearelondonmade.com/tech/driving-assistant/ 
Configured with: /mnt/share/cygpkgs/gcc/gcc.x86_64/src/gcc-10.2.0/configure
--srcdir=/mnt/share/cygpkgs/gcc/gcc.x86_64/src/gcc-10.2.0 --prefix=/usr
--exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc
--docdir=/usr/share/doc/gcc --
https://waytowhatsnext.com/computers/discord-and-steam/
htmldir=/usr/share/doc/gcc/html -C --build=x86_64-pc-cygwin
--host=x86_64-pc-cygwin --target=x86_64-pc-cygwin --without-libiconv-prefix
--without-libintl-prefix --
http://www.iu-bloomington.com/property/properties-in-turkey/
libexecdir=/usr/lib --with-gcc-major-version-only --enable-shared
--enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs
--enable-bootstrap --enable-__cxa_atexit --with-dwarf2
https://komiya-dental.com/sports/telegram/ --with-tune=generic
--enable-languages=c,c++,fortran,lto,objc,obj-c++ --enable-graphite
--enable-threads=posix --enable-libatomic --enable-libgomp --enable-libquadmath
http://www-look-4.com/health/winter-sickness/ --enable-libquadmath-support
--disable-libssp --enable-libada --disable-symvers --with-gnu-ld --with-gnu-as
--with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix
--without-libintl-prefix --with-system-zlib
https://www.webb-dev.co.uk/sports/gym-during-covid/ --enable-linker-build-id
--with-default-libstdcxx-abi=gcc4-compatible --enable-libstdcxx-filesystem-ts
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.0 (GCC)
Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.