New: ext4 file system corruption

https://bugzilla.kernel.org/show_bug.cgi?id=201685

            Bug ID: 201685
           Summary: ext4 file system corruption
           Product: File System
           Version: 2.5
    Kernel Version: maybe one of 4.18.18 4.19.1 4.20-rc2
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 279431
--> https://bugzilla.kernel.org/attachment.cgi?id=279431&action=edit
dmesg 4.18.18 amdgpu.dc=0

My system was fine when I shut it down on Sunday Nov 11. Today Nov 13 I booted 4.19.1, built two new kernels 4.20-rc2 4.18.18 (using a tmpfs, not the SSD or HDD), then booted into those kernels briefly (to test if a different bug had been fixed). Finally I booted into 4.18.18 (setting amdgpu.dc=0 to workaround my other bug), and after some moments experienced symptoms of filesystem corruption on opening an xterm:

sed: error while loading shared libraries:
/lib/x86_64-linux-gnu/libattr.so.1: unexpected PLT reloc type 0x00000107
sed: error while loading shared libraries:
/lib/x86_64-linux-gnu/libattr.so.1: unexpected PLT reloc type 0x00000107
claude@eiskaffee:~$

I fixed it by extracting the relevant file from the Debian archive on a different machine and using cat with bash shell IO redirection to overwrite the corrupted shared library file on my problem machine.

Here are the relevant versions extracted from my syslog:

    Nov 13 15:45:49 eiskaffee kernel: [    0.000000] Linux version 4.19.1
(claude@eiskaffee) (gcc version 8.2.0 (Debian 8.2.0-9)) #1 SMP Tue Nov 6
14:58:04 GMT 2018
    Nov 13 18:44:12 eiskaffee kernel: [    0.000000] Linux version 4.20.0-rc2
(claude@eiskaffee) (gcc version 8.2.0 (Debian 8.2.0-9)) #1 SMP Tue Nov 13
16:38:55 GMT 2018
    Nov 13 18:45:00 eiskaffee kernel: [    0.000000] Linux version 4.18.18
(claude@eiskaffee) (gcc version 8.2.0 (Debian 8.2.0-9)) #1 SMP Tue Nov 13
16:23:11 GMT 2018
    Nov 13 18:46:13 eiskaffee kernel: [    0.000000] Linux version 4.18.18

(claude@eiskaffee) (gcc version 8.2.0 (Debian 8.2.0-9)) #1 SMP Tue Nov 13 16:23:11 GMT 2018

mount says:

/dev/nvme0n1p2 on / type ext4 (rw,relatime,errors=remount-ro)

The machine in question is my production workstation, so I don't feel like testing anything that might result in data loss.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Jason Gambrel ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #1 from Jason Gambrel ([email protected]) --- Similar problems on Linux Mint 19 Tara using kernels 4.19.0 and 4.19.1

On rebooting my ASUS UX 430U laptop I ended up at the initramfs prompt and had to run fsck to repair the root file system. I was then able to continue booting. Sorry didn't save the log files. This has happened randomly twice. I will post them if this happens again.

Did not happen under 4.18

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Theodore Tso ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #2 from Theodore Tso ([email protected]) ---
I'm using a 4.19.0 based kernel (with some ext4 patches for the 4.20 mainline)
and I'm not noticing any file system problems. I'm running a Dell XPS 13 with
an NVME SSD, and Debian testing as my userspace.

It's hard to do anything with a "my file system is corrupted" report without
any kind of reliable reproduction information. Remember that file system
corruptions can be caused by any number of things --- buggy device drivers, buggy Nvidia binary modules that dereference wild pointers and randomly corrupt kernel memory, RAID code if you are using RAID, etc., etc.

Also, the symptoms reported by Claude and Jason are very different. Claude has reported that a data block in a shared library file has gotten corrupted.
Jason has reported that file system metadata corruption. This could very well
be coming from different root causes.

So it's better with these sorts of things to file separate bugs, and to include detailed hardware configuration details, kernel configuration, dumpe2fs outputs of the file system in question, as well as e2fsck logs.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #3 from Jason Gambrel ([email protected]) --- (In reply to Theodore Tso from comment #2)

Thank you for your reply Theodore and I apologize for my unhelpful post. I am relatively new to this space so I find your advice very helpful. If the file system corruption happens a 3rd time (hopefully it won't), I will post a separate bug report. I also wasn't previously aware of dumpe2fs, so I will provide that helpful information next time. I have also searched to find any additional logs and it looks like fsck logs the boot info under /var/logs/boot.log and potentially /var/logs/syslog. Unfortunately the information from my last boot requiring fixation had already been overwritten. I will keep this in mind for the future.

If it helps, my system uses an i7 with integrated Intel graphics. I am not running any proprietary drivers. No Raid. 500gb SSD. 16gb ram with a 4gb swap file (not a swap partition). I have been using ukuu to install mainline kernels. I did not change anything else on my system. When I jumped from 4.18.17 to 4.19.0 this problem first appeared. Then it occurred again after updating to 4.19.1.

I'm uncertain as to whether it would be helpful or not, but while trying to figure out why this happened to me, I came across a post on Ask Ubuntu with a few others reporting similar problems. They did provide some debugging information in their post at:
https://askubuntu.com/questions/1092558/ubuntu-18-04-4-19-1-kernel-after-closing-the-lid-for-the-night-not-logging-ou

Again it might be a different problem from what Claude Heiland-Allen is experiencing.

Thank you very much for your advice and I will try and provide some useful information including logs in a separate bug report if it happens again.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #4 from Theodore Tso ([email protected]) ---
Thanks for pointing out that bug. I'll note that the poster who
authoritatively claimed that 4.19 is safe, and the bug obviously was introduced
in 4.19.1 didn't bother to do a "git log --stat v4.19 v4.19.1". This would
show that the changes were all in the Sparc architecture support, networking drivers, the networking stack, and a one-line change in the crypto subsystem....

This is why I always tell users to report symptoms, not diagnosis. And for
sure, not to bias their observations by their their certainty that they have
diagnosed the problem. (If they think they have diagnosed the problem, send
me a patch, preferably with a reliable repro so we can add a regression test. :-)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #5 from [email protected] ---
Two of my Linux machines experience regular ext4 file system corruption since I updated them to 4.19. 4.18 was fine. I noticed the problem first when certain file operations returned "Structure needs cleaning". fsck then mostly finds dangling inodes (of files I have written recently), incorrect reference counts, and so on. Both machines do not use RAID, don't use any proprietary drivers and both have an Intel board. One of them uses an SSD and one of them a HDD. Unfortunately, I don't know which information might be useful to you.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #6 from Theodore Tso ([email protected]) ---
Please send detailed information about your hardware (lspci -v and dmesg while
it is booting would be helpful). Also please send the results of running
dumpe2fs on the file system, and the kernel logs when file system operations
started returning "Structure needs cleaning". I want to see if there are any
other kernel messages in and around the ext4 error messages that will be in the
kernel logs. Also please send me the fsck logs, and what sort of workload
(what programs) you have running on your system. Also, do you do anything
unusual on your machine; do you typically do clean shutdowns, or do you just do
forced power-offs? Are you regularly running into a large amount of memory
pressure (e.g., are you regularly using a large percentage of the physical memory available on your system.)

This is going to end up being a process of elimination. 4.19 works for me.
I'm using a 2018 Dell XPS 13, Model 9370, with 16GB of memory and I run a
typical kernel developer workload. We also run a large number of ext4
regression testing, which generally happens on KVM for one developer, and I use
Google Compute Engine for my tests. None of this detected any problems before
4.19 was released. So the question then is --- what makes people who are
experiencing difficulties different from my development laptop (which also has an Intel board, and an SSD connected using NVMe) from those who are seeing problems? This is why getting lots of details about the precise hardware configuration is going to be critically important.

In the ideal world we would come up with a clean, simple, reliable reproducer. Then we can experiment and see if the reliable reproducer continues to reproduce on different hardware, etc.

Finally, since in order to figure things out we may need a lot of detail about the hardware, the software, and the applications running on each of the systems where people are seeing problems, it's helpful if new people upload all of this information onto new kernel bugzilla issues, and then mention the kernel bugzilla issue here, so people can follow the links.

I'll note that a few years ago, we had a mysterious "ext4 failure" that ultimately turned out to be a Intel virtualization hardware bug, and it was the *host* version that mattered, not the *guest* kernel version that mattered. Worse, it was fixed in the very next vesion of the kernel, and so it was only people using Debian host kernels that ran into troubles --- but **only** if
they were using a specific Intel chipset and Intel CPU generation. Everyone
kept on swearing up and down it was an ext4 bug, and there were many angry
people arguing this on bugzilla. Ultimately, it was a problem caused by a
hardware bug, and a kernel workaround that was in 3.18 but not in 3.17, and
Debian hadn't noticed they needed to backport the kernel workaround.... And
because everyone was *certain* that the host kernel version didn't matter --- after all, it was *obviously* an ext4 bug in the guest kernel --- they didn't report it, and that made figuring out what the problem was (it took over a year) much, Much, MUCH harder.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #7 from [email protected] ---
Confirming for now, similar problems with 4.19.1, 4.19.2, 4.20-rc1, 4.20-rc2 and 4.20-rc3 from the Ubuntu mainline-kernel repository
(http://kernel.ubuntu.com/~kernel-ppa/mainline/).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #8 from [email protected] ---
I could reproduce the issue. Probably not relevant but I had to modify the initramfs script, 4.19.0 kernel for any reason changed the mdp raid major number from 245 to 9 (i.e on a devtmpfs filesystem) and renamed them /dev/mdX instead of /dev/md_dX as before. Partitions are now major 259. Also the md/lvm devices became faster by the way. I tried ext4 mmp protection but without success (i.e not a multi mount issue). 4.20.0-rc3 kernel gives the same sort of issue. It is reproducible on an other amd64 machine with the same configuration but different hardware.

Applications used for the test,
sys-fs/lvm2-2.02.173, sys-fs/mdadm-4.1, sys-fs/e2fsprogs-1.44.4, sys-apps/coreutils-8.30, sys-apps/util-linux-2.33

More info in ext4_iget.txt.xz

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #9 from [email protected] ---
Created attachment 279557
--> https://bugzilla.kernel.org/attachment.cgi?id=279557&action=edit
more infos (lspci, dmesg, etc.)

compressed text file

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #10 from [email protected] ---
(In reply to Jimmy.Jazz from comment #8)

typo: read -rc2 not -rc3

I'm using rc3 release now. /etc/mke2fs.conf default_mntopts user_xattr is deactivated (tune2fs -o ^user_xattr /dev/mapper/xx) on all my lvm devices. Native mdp devices still have the option set.
One of my machines is always on heavy load because of daily compilations basis I do in a nilfs sandbox environment. No error for the moment.

The issue was all of sudden and affected all my lvm devices. Next reboot, fsck randomly couldn't "see" any failure the kernel had reported but detection improved when used with the -D optimization option.

It could be some old corruptions undetected until now. One of the server is more then 5 years old without reinstall but still with regular updates. But the other one still has the issue at its first installation.

The kernel was compiled with GCC and LD=ld.bfd
I was unsuccessful with CLANG.

# gcc --version
gcc (Gentoo 8.2.0-r4 p1.5) 8.2.0

I'm bit puzzled.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #11 from Theodore Tso ([email protected]) ---
Thanks Jimmy for your report. Can you specify what sort of LVM devices are you
using? Is it just a standard LVM volume (e.g., no LVM raid, no LVM snapshops,
no dm-thin provisioning)?

The reason why I ask is because I've run gce-xfstests on 4.19, 4.19.1, and 4.19.2, and it uses LVM (nothing fancy just standard LVM volumes, although xfstests will layer some dm-error and dm-thin on top of the LVM volumes for specific xfstests) on top of virtio-scsi on top of Google Compute Engine's Persistent Disks, and I'm not noticing any problems.

I just noticed that my .config file for my GCE testing has CONFIG_SCSI_MQ_DEFAULT set to "no", which means I'm not using the new block-mq
data path. So perhaps this is a MQ specific bug? (Checking... hmm, my
laptop running 4.19.0 plus the ext4 commits landing in 4.20-rc2+ is *also*
using CONFIG_SCSI_MQ_DEFAULT=n.) And Kconfig recommends that SCSI_MQ_DEFAULT
be defaulted to y.

This is why having people include their Kernel configs, and what devices they
use is so important. The vast amount of time, given the constant testing
which we do in the ext4 layer, more often than not the problem is somewhere
*else* in the storage stack. There have been bugs which have escaped notice
by our tests, yes. But it's rare, and it's almost never the case when a large number of users are reporting the same problem.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Claude Heiland-Allen ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

 Attachment #279431|0                           |1

is obsolete| |

--- Comment #12 from Claude Heiland-Allen ([email protected]) --- Created attachment 279569
--> https://bugzilla.kernel.org/attachment.cgi?id=279569&action=edit
eiskaffee - logs and other info

I replaced my single dmesg attached with further information (as requested) in a tarball. Contains kernel configs, dmesg logs, tune2fs -l, lspci -vvv. I don't use LVM or MD on my machine (eiskaffee). The file system corruption I experienced was on the root partition on SSD.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #13 from [email protected] ---
(In reply to Theodore Tso from comment #11)

Standard lvm linear volumes on top of a full /dev/md0p5 pv partition on both machines with respectively kernel 4.19.2 and 4.20.0-rc3

CONFIG_SCSI_MQ_DEFAULT=y on both machines
CONFIG_DM_MQ_DEFAULT is not set

sorry it was an oversight. See attachments.

server with kernel 4.19.2

# smartctl -i /dev/sdb                                                 
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.2-radeon] (local build)

=== START OF INFORMATION SECTION ===

Device Model:     MKNSSDRE512GB
Serial Number:    MK15090210005157A
LU WWN Device Id: 5 888914 10005157a
Firmware Version: N1007C
User Capacity:    512 110 190 592 bytes [512 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Nov 21 19:02:07 2018 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

# smartctl -i /dev/sda                                                 
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.2-radeon] (local build)

=== START OF INFORMATION SECTION ===

Device Model:     MKNSSDRE512GB
Serial Number:    MK150902100051556
LU WWN Device Id: 5 888914 100051556
Firmware Version: N1007C
User Capacity:    512 110 190 592 bytes [512 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Nov 21 19:04:01 2018 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

server with 4.20.0-rc3

# smartctl -i /dev/sda                                                
=== START OF INFORMATION SECTION ===
Model Family:     HGST Travelstar 7K1000
Device Model:     HGST HTS721010A9E630
Serial Number:    JR10004M0BD4YF
LU WWN Device Id: 5 000cca 8a8c52dba
Firmware Version: JB0OA3J0
User Capacity:    1 000 204 886 016 bytes [1,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Nov 21 19:02:21 2018 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

I do periodic backup and the strange is dm-7 is only accessed read only and it just triggered ext4-iget failures.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #14 from [email protected] ---
Created attachment 279571
--> https://bugzilla.kernel.org/attachment.cgi?id=279571&action=edit
dm-7 device w/ bad extra_isize errors

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #15 from [email protected] ---
Created attachment 279573
--> https://bugzilla.kernel.org/attachment.cgi?id=279573&action=edit
dmesg w/ EXT4-fs error only

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #16 from [email protected] ---
Created attachment 279575
--> https://bugzilla.kernel.org/attachment.cgi?id=279575&action=edit
.config of 4.19.2

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #17 from [email protected] ---
Created attachment 279577
--> https://bugzilla.kernel.org/attachment.cgi?id=279577&action=edit
.config of 4.20.0-rc3

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #18 from [email protected] ---
kernel 4.19.2
md0 (formerly md_d0)

is a raid1 GPT bootable device

md0 : active raid1 sda[0] sdb[1]

      499976512 blocks super 1.2 [2/2] [UU]
      bitmap: 3/4 pages [12KB], 65536KB chunk

md0p1 is the grub boot partition

# fdisk -l /dev/md0                                                    
Disque /dev/md0 : 476,8 GiB, 511975948288 octets, 999953024 secteurs
Unités : secteur de 1 × 512 = 512 octets
Taille de secteur (logique / physique) : 512 octets / 512 octets
taille d'E/S (minimale / optimale) : 512 octets / 512 octets
Type d'étiquette de disque : gpt
Identifiant de disque : 9A6C46CD-3B9C-4C64-AE3C-EDB416548134

Périphérique Début Fin Secteurs Taille Type
/dev/md0p1 40 2088 2049 1M Amorçage BIOS
/dev/md0p2 2096 264240 262145 128M Système de fichiers Linux

/dev/md0p3    264248   2361400   2097153     1G Système de fichiers Linux
/dev/md0p4   2361408   6555712   4194305     2G Système de fichiers Linux

/dev/md0p5 6555720 999952984 993397265 473,7G Système de fichiers Linux

# fdisk -l /dev/sdb                                                    
Disque /dev/sdb : 477 GiB, 512110190592 octets, 1000215216 secteurs
Modèle de disque : MKNSSDRE512GB   
Unités : secteur de 1 × 512 = 512 octets
Taille de secteur (logique / physique) : 512 octets / 512 octets
taille d'E/S (minimale / optimale) : 512 octets / 512 octets
Type d'étiquette de disque : dos
Identifiant de disque : 0x58e9a5ac

Périphérique Amorçage Début Fin Secteurs Taille Id Type
/dev/sdb1 8 1000215215 1000215208 477G fd RAID Linux
autodétec

idem for second computer same configuration but one HGST Travelstar 7K1000 disk attached and kernel cmdline has mdraid=forced

If I missed something just let me know.

Thx for your help.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #19 from [email protected] ---
(In reply to Theodore Tso from comment #11)

So perhaps this is a MQ specific bug?

I checked old .config and I had CONFIG_SCSI_MQ_DEFAULT=y activated since version 4.1.6.

MQ investigation will probably lead us to a dead end.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #20 from Theodore Tso ([email protected]) ---
Can someone try 4.19.3? I was working with another Ubuntu user who did *not*
have see the problem with 4.19.0, but did see it with 4.19.1, but one of the differences in his config was:

-# CONFIG_SCSI_MQ_DEFAULT is not set
+CONFIG_SCSI_MQ_DEFAULT=y

Furthermore, he tried 4.19.3 and after two hours of heavy I/O, he's no longer
seeing problems. Based on the above observation, his theory is this commit
may have fixed things, and it *is* blk-mq specific:

commit 410306a0f2baa5d68970cdcf6763d79c16df5f23

Author: Ming Lei <[email protected]>
Date:   Wed Nov 14 16:25:51 2018 +0800

SCSI: fix queue cleanup race before queue initialization is done

commit 8dc765d438f1e42b3e8227b3b09fad7d73f4ec9a upstream.

c2856ae2f315d ("blk-mq: quiesce queue before freeing queue") has already fixed this race, however the implied synchronize_rcu() in blk_mq_quiesce_queue() can slow down LUN probe a lot, so caused performance regression.

Then 1311326cf4755c7 ("blk-mq: avoid to synchronize rcu inside
blk_cleanup_queue()")
tried to quiesce queue for avoiding unnecessary synchronize_rcu() only when queue initialization is done, because it is usual to see lots of inexistent LUNs which need to be probed.

However, turns out it isn't safe to quiesce queue only when queue initialization is done. Because when one SCSI command is completed, the user of sending command can be waken up immediately, then the scsi device may be removed, meantime the run queue in scsi_end_request() is still in-progress, so kernel panic can be caused.

In Red Hat QE lab, there are several reports about this kind of kernel panic triggered during kernel booting.

This patch tries to address the issue by grabing one queue usage counter during freeing one request and the following run queue.

This commit just landed in mainline and is not in 4.20-rc2, so the theory that it was a blk-mq bug that was fixed by the above commit is consistent with all of the observations made to date.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #21 from Claude Heiland-Allen ([email protected]) --- My kernels have this:

    4.18.19.config:CONFIG_SCSI_MQ_DEFAULT=y
    4.19.2.config:CONFIG_SCSI_MQ_DEFAULT=y
    4.20-rc3.config:CONFIG_SCSI_MQ_DEFAULT=y

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #22 from [email protected] ---
I will ... if I can reboot safely. This time it affects / (i.e /dev/md0p3) What a nightmare.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #23 from Theodore Tso ([email protected]) ---
Jimmy, I don't blame you. Unfortunately, I don't have a clean repro of the
problem because when I tried building a 4.20-rc2 kernel with CONFIG_SCSI_MQ_DEFAULT=y, and tried running gce-xfstests, no problems were detected. And I'm too chicken to try running a kernel version which does have the problem reported with CONFIG_SCSI_MQ_DEFAULT=y on my primary development
laptop. :-)

I will say that if you are seeing problems on a particular file system (e.g. /), by the time the kernel is reporting inconsistencies, the damage is already done. Yes, you might want to try doing a backup before you reboot, in case the system doesn't come back, but realistically speaking, the longer you keep running, the problems are more likely to compound.

So from a personally very selfish perspective, I'm hoping someone who has already suffered corruption problems is willing to try either 4.19.3, or disabling CONFIG_SCSI_MQ_DEFAULT, or both, and report that they are no longer seeing problems, than my putting my own personal data at risk....

Maybe over T-day weekend, I'll try doing a full backup, and then try using 4.19.3 on my personal laptop --- but a "it works fine for me" report won't necessarily mean anything, since to date I'm not able to reproduce the problem on one of my systems.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Jens Axboe ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #24 from Jens Axboe ([email protected]) ---
It'd be critical to know if 4.19.3 is still showing the issue with MQ being on. I'm going to try my luck at reproducing this issue as well, but given that there hasn't been a lot of noise about it, not sure I'll have too much luck.

I've got a few suspects, so I'm also willing to spin a patch against 4.19.3 if folks are willing to give that a go.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #25 from Jens Axboe ([email protected]) ---
Ted, it seems to be affecting nvme as well, so there's really no escaping for you. But there has to be some other deciding factor here, or all the block testing would surely have caught this. Question is just what it is.

What would be the most helpful is if someone who can reproduce this at well could run a bisect between 4.18 and 4.19 to figure out wtf is going on here. This commit:

commit 410306a0f2baa5d68970cdcf6763d79c16df5f23

Author: Ming Lei <[email protected]>
Date:   Wed Nov 14 16:25:51 2018 +0800

SCSI: fix queue cleanup race before queue initialization is done

might explain the SCSI issues seen, but the very first comment is from someone using nvme that the above patch has no bearing on that at all. It is, however, possible that some of the queue sync patches caused a blk-mq issue, and that is why nvme is affected as well, and why the above commit seems to fix things on the SCSI side.

I'm going to attach a patch here for 4.19 and it'd be great if folks could try that.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #26 from Jens Axboe ([email protected]) --- Created attachment 279579
--> https://bugzilla.kernel.org/attachment.cgi?id=279579&action=edit
4.19 patch

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #27 from Jens Axboe ([email protected]) --- Created attachment 279581
--> https://bugzilla.kernel.org/attachment.cgi?id=279581&action=edit
4.20-rc3 patch

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #28 from Jens Axboe ([email protected]) ---
If it's not this, another hint might be a discard change. Is everyone affected using discard?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #29 from [email protected] ---
Sorry for the late response, but I have been trying to reproduce the problem with 4.19.2 for some while now. It seems that the problem I was experiencing only happens with 4.19.1 and 4.19.0, and it did so very frequently. I can at least confirm that I have CONFIG_SCSI_MQ_DEFAULT=y set in 4.19 but I didn't in 4.18. I hope that this is, at least for me, fixed for now.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Henrique Rodrigues ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected].
                   |                            |pt

--- Comment #30 from Henrique Rodrigues ([email protected]) --- (In reply to Theodore Tso from comment #23)

On a Ubuntu 18.10 machine I've upgraded to 4.19.0 and started getting these corruption errors. Yesterday I've upgraded to 4.19.3 and was still getting corrupted. 4.18 was fine. Unfortunately the latest corruption rendered the operating system unbootable. I'm going to try and fix it tonight and then will try to disable CONFIG_SCSI_MQ_DEFAULT and test.

I'm in a somewhat fortunate position since the data that I care about lives on another disk with a different filesystem type, so the corruption on the root filesystem is just annoying and not really that dangerous.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Azat Khuzhin ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #31 from Azat Khuzhin ([email protected]) --- I'm also running 4.20-rc2 but does not experience any corruption for now (*crossing fingers*)

(In reply to Theodore Tso from comment #11)

But I do have CONFIG_SCSI_MQ_DEFAULT:

$ zgrep CONFIG_SCSI_MQ_DEFAULT=y /proc/config.gz
CONFIG_SCSI_MQ_DEFAULT=y
$ head /sys/block/dm-0/dm/use_blk_mq
1

(In reply to Jens Axboe from comment #25)

And I do have nvme ssd:

nvme0n1 259:0 0 953.9G 0 disk

├─nvme0n1p1   259:1    0   260M  0 part  /boot
└─nvme0n1p2   259:2    0 953.6G  0 part
  └─cryptroot 254:0    0 953.6G  0 crypt /

And as you can see I have dm-crypt

So it looks like that this is not that simple (IOW not every setup/env/hw affected).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #32 from Azat Khuzhin ([email protected]) --- (In reply to Jens Axboe from comment #28)

And what a coincidence, before upgrading to 4.20-rc2 I enabled discard:

# findmnt /
TARGET SOURCE                FSTYPE OPTIONS
/      /dev/mapper/cryptroot ext4   rw,relatime,discard

# cat /proc/cmdline
cryptdevice=...:cryptroot:allow-discards

# cryptsetup status cryptroot
/dev/mapper/cryptroot is active and is in use.
  ...
  flags:   discards

Plus I triggered fstrim manually at start:

# systemctl status fstrim
Nov 19 00:16:14 azat fstrim[23944]: /boot: 122.8 MiB (128716800 bytes) trimmed
on /dev/nvme0n1p1
Nov 19 00:16:14 azat fstrim[23944]: /: 0 B (0 bytes) trimmed on
/dev/mapper/cryptroot

But what is interesting here is that it did not do any discard for the "/", hm (does ext4 did it for me at start?)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #33 from [email protected] ---
(In reply to Jens Axboe from comment #28)

All 'cat /sys/block/dm-*/dm/use_blk_mq' are zero. Could MQ still be a suspect ?

I reproduced the issue with 4.19.3 as well but without your patch. The difference is, it happens less often but still under heavy load (hours of work, mostly compilations and monitoring).

The strange is, the affected disks are not obliged to be under load and on the next reboot fsck -f show some of them as clean despite they were declared with ext4_iget corruptions (tested during reboot from 4.19.2 to 4.19.3 kernel)!

It's like some shared fs cache failure to me with unpleasant consequences.

Disabling user_xattr seems to be more helpful with 4.20.0-rc3 anyway, no error since. Actually not under heavy load.

Failures appear also on an plain old HDD device. For me, SSD discard is more a consequence as a reason but it's worse investigating it.

I will try your patch ASAP.

Thx

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Bart Van Assche ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #34 from Bart Van Assche ([email protected]) ---
I hit filesystem corruption with a desktop system running openSUSE Tumbleweed, kernel v4.19.3 and ext4 on top of a SATA SSD with scsi_mod.use_blk_mq=Y in /proc/cmdline. Discard was not enabled in /etc/fstab. After having enabled fsck.mode=force the following appeared in the system log after a reboot:

/dev/sda2: Inode 12190197 extent tree (at level 2) could be narrower. IGNORED.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Reindl Harald ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #35 from Reindl Harald ([email protected]) ---

that is completly unrelated, i see that for years now on several machines and not cleaned up automatically and wasting my time to boot in rescure mode is not worth given the low importance of "could"

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #36 from Bart Van Assche ([email protected]) --- (In reply to Reindl Harald from comment #35)

That's good to know. The reason I commented on this bug report and that I replied that I hit data corruption is because my workstation failed to boot due to fsck not being able to repair the file system automatically. I had to run fsck manually, answer a long list of scary questions and reboot.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #37 from [email protected] ---
@Jen Axboe

please read worth not worse in comment 33

I tried your patch for 4.19.3 and still get quite harmful ext4 errors like this one,

EXT4-fs error (device dm-4): ext4_xattr_ibody_get:592: inode #4881425: comm
rsync: corrupted in-inode xattr

The filesystems were clean at boot time and the system was idle.

tune2fs ends with

FS Error count:           64
First error time:         Fri Nov 23 00:19:25 2018
First error function:     ext4_xattr_ibody_get
First error line #:       592
First error inode #:      4881425
First error block #:      0
Last error time:          Fri Nov 23 00:19:25 2018
Last error function:      ext4_xattr_ibody_get
Last error line #:        592
Last error inode #:       4881430
Last error block #:       0
MMP block number:         9255
MMP update interval:      5

If you are interested in its dumpe2fs result let me know.

I don't use binary modules.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #38 from Rainer Fiebig ([email protected]) --- (In reply to Jimmy.Jazz from comment #37)

Jimmy,

what *I* would do if I were in your shoes is

- run a kernel < 4.19, make sure the fs is OK and *backup important data*
- compile 4.19.3 with CONFIG_SCSI_MQ_DEFAULT *not* set

and see what happens. If you still get corruption, CONFIG_SCSI_MQ_DEFAULT probably is not the culprit. If not, it has at least something to do with it.

It seems that CONFIG_SCSI_MQ_DEFAULT *not* set was the default <= 4.18.19.

Others here obviously don't have problems with CONFIG_SCSI_MQ_DEFAULT=y and kernels >= 4.19, but you never know.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

AdamB ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #39 from AdamB ([email protected]) ---
I also experienced an ext4 file system corruption with 4.19.1, after resuming from suspend-to-ram.

I've ran 4.12, 13, 14, 16, 17, and 18 on the same machine with near identical .config and never had a file system corruption.

For all those kernels, I've had CONFIG_SCSI_MQ_DEFAULT=y.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #40 from Rainer Fiebig ([email protected]) --- (In reply to AdamB from comment #39)

I can say the same for CONFIG_SCSI_MQ_DEFAULT=n. But this is not exactly the same as running 4.19.x with CONFIG_SCSI_MQ_DEFAULT=n.

As someone already pointed out: the best way to find out what's behind this is bisecting between 4.18 and 4.19 by someone affected by the problem. This is time consuming but in the end may also be the fastest way. A backup is IMO mandatory in this case.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

HB ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected]
                   |                            |.de

--- Comment #41 from HB ([email protected]) --- I have this bug with ubuntu 18.04 kernel 4.15.0-39, too.

My Desktop:
SSD (Samsung 840) with three partions:
/boot : ext2
/ : ext4
swap

HDD1: one ext4 partition
HDD2: luks encrypted, never mounted at boot time and not used when the error happens.

No Raid-Stuff used.

The problems only occurs on the ext4 part. from the ssd.

Sometimes at booting there are some message like "could not access ata devices", there are some timeouts with ATA-commands. It retries several times until it gives up, I dont reach the busy box command line.

Sometimes I reach the busybox command line but cant fix it there, because there is no fsck in busybox.

I have to connect the ssh to a notebook via usb2sata Adapter, the two partions were recognised without problems and are in most cases automounted. If I force a fsck there are some orphaned inodes discovered and the fs is fixed.

After this I can boot from this SSD in the desktop without problems until it happens again.

The weired thing is that sometimes the SSD is not recognised after this and has this ATA-Timeouts above.

Even turning the desktop completeley powerless (disconnecting from power socket and waiting some minutes then doing a cold boot) it gets stuck the same way.

On the notbook were I fix the SSD is the same OS installed and there never occured this type of problem. Maybe I had only luck until now, I dont use the notebook very much.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #42 from HB ([email protected]) ---
add: I (In reply to HB from comment #41)

I mean the "SSD" not ssh.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #43 from [email protected] ---
I do some bisecting on the linux-master git source.
I'm at the kernel version 4.19.0-rc2-radeon-00922-gf48097d294-dirty currently. I hope that all the fscks didn't make my system immune to this issue :)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #44 from Jens Axboe ([email protected]) ---
Thanks a lot, Jimmy! That's what we need to make some progress here, in lieu of me and/or Ted being able to reproduce this issue.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #45 from Henrique Rodrigues ([email protected]) --- (In reply to Rainer Fiebig from comment #38)

This is what I did: recompile my 4.19.3 kernel with CONFIG_SCSI_MQ_DEFAULT=n. I've used the computer normally, ran some heavy read/write operations, rebooted a bunch of times and had no problems since then. I'm on Ubuntu 18.10 with an SSD, encrypted LUKS root partition.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #46 from Theodore Tso ([email protected]) ---
So Henrique, the only difference between the 4.19.3 kernel that worked and the
one where you didn't see corruption was CONFIG_SCSI_MQ_DEFAULT? Can you diff
the two configs to be sure?

What can you tell us about the SSD? Is it a SATA-attached SSD, or NVMe-attached?

What I can report is my personal development laptop is running 4.19.0 (plus the ext4 patches that landed in 4.20-rc1) with CONFIG_SCSI_MQ_DEFAULT=n? (Although as others have pointed out, that shouldn't matter since my SSD is
NVMe-attached, and so it doesn't go through the SCSI stack.) My laptop runs
Debian unstable, and uses an encrypted LUKS partition on top of which I use
LVM. I do use regular suspend-to-ram (not suspend-to-idle, since that burns
way too much power; there's a kernel BZ open on that issue) since it is a laptop.

I have also run xfstest runs using 4.19.0, 4.19.1, 4.19.2, and 4.20-rc2 with CONFIG_SCSI_MQ_DEFAULT=n; it's using the gce-xfstests[1] test appliance which means I'm using virtio-SCSI on top of LVM, and it runs a large number of regression tests, many with heavy read/write loads, but none of the file systems is mounted for more than 5-6 minutes before we unmount and then run fsck on it. We do *not* do any suspend/resumes, although we do test the file system side of suspend/resume using the freeze and thaw ioctls. There were no unusual problems noticed.

[1] https://thunk.org/gce-xfstests

I have also run gce-xfstests on 4.20-rc2 with CONFIG_SCSI_MQ_DEFAULT=y, with
the same configuration as above --- vrtio-scsi with LVM on top. There was
nothing unusual that was detected there.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #47 from Theodore Tso ([email protected]) --- Bart, in #34, was the only thing which e2fsck reported this:

/dev/sda2: Inode 12190197 extent tree (at level 2) could be narrower. IGNORED.

That's not a file system problem; it's a potential optimization which e2fsck detected, which would eliminate a random 4k read when running random read workload against that inode. If you don't want to see this, you can use e2fsck's "-E fixes_only" option.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #48 from Rainer Fiebig ([email protected]) --- (In reply to Jimmy.Jazz from comment #43)

Great! Everyone who's had his share of bisecting knows to value you effort! ;)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #49 from [email protected] ---
to be short,

Release 4.19.0-rc2-radeon-00922-gf48097d294-dirty

A: Nothing append at first when the computer is nearly idle.

B: I mounted an usb SD media first ro (default) then rw. Transfer to it some big files (cp and tar) from two different xterms. Lot of errors, stick became for the kernel read only. Transfer failed.
Umount then remount the filesystem without doing an fsck and restart the transfer again.
Transfer ok.
umount ok.

I will next declare it git bisect bad and then reboot.

please see attachements.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #50 from [email protected] ---
Created attachment 279655
--> https://bugzilla.kernel.org/attachment.cgi?id=279655&action=edit
dmesg EXT4 errors

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #51 from [email protected] ---
Created attachment 279657
--> https://bugzilla.kernel.org/attachment.cgi?id=279657&action=edit
dumpe2fs

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #52 from [email protected] ---
FYI, I need 2 patches for my initramfs to generate and IMO should not interfere.

drivers/tty/vt/defkeymap.map to get the fr kbd mapping usr/Makefile due to shell evaluation

@@ -43,7 +43,7 @@
 targets := $(datafile_y)
 
 # do not try to update files included in initramfs
-$(deps_initramfs): ;
+$(deps_initramfs): ; 
 
 $(deps_initramfs): klibcdirs
 # We rebuild initramfs_data.cpio if:

@@ -52,5 +52,6 @@
 # 3) If gen_init_cpio are newer than initramfs_data.cpio
 # 4) arguments to gen_initramfs.sh changes
 $(obj)/$(datafile_y): $(obj)/gen_init_cpio $(deps_initramfs) klibcdirs
-       $(Q)$(initramfs) -l $(ramfs-input) > $(obj)/$(datafile_d_y)
+       $(Q)$(initramfs) -l $(ramfs-input) | \
+       sed '2,$$s/:/\\:/g' > $(obj)/$(datafile_d_y)
        $(call if_changed,initfs)

[quote]
I don't want to break T.Tso rules, but I remember, I have encountered a similar issue when I initially tried partitionable array with major 9. At that time I switched to major 254 as explain in comment 8 and the problem didn't come up since... until the recent kernel 4.19 with mdadm 4.1 and kernel devtmpfs that switched the metadevices to major 9. Also, why? A big mystery. [/quote]

Now to the fact,
I was able to reboot in rescue mode, I use the world service to illustrate the process. Nothing to do with Debian.

# service mdraid start
# service vg0 start
# cd /dev/mapper
# for i in *-*; do fsck /dev/mapper/$i; done
All clean except sys-scm (f  word)
the usb stick is clean too.
I need a terminal for interactive repairs so I write the beginning by hand.

Inode 58577 has extra size (103) which is invalid
Fix<y>? yes
Timestamp(s) on inode 58577 beyond 2310-04-04 are likely pre-1970
+ 9 others
Inodes that were part of corrupted orphan linked list found. Fix<y>?yes
+ 3 others
i_size is 139685221367808, shoud be 0.
i_blocks is 32523, should be 0.
+ 22 others
Pass 2: checking directory structure
Inode 58577 (/git/toolkit.git/objects/e0) has invalid mode (0150)
+ 9 others
Unattached inode 17013
Connect to /lost+found
Inode 17013 ref count is 2, should be 1
+ 35 others
Inode 58586 (...) has invalid mode (0122)
+ 5 others
[...]
Unattached inode 262220
Connect to /lost+found<y>? yes
Inode 262220 ref count is 2, should be 1. Fix<y>? yes
Pass 5: Checking group summary information
Block bitmap differences: -(9252--9255) -10490 -(10577--10578) -(16585--16589) -295391 -(682164--682165)
Fix<y>? yes
Free blocks count wrong for group #0 (2756, counted=2768).
Fix<y>?
Block bitmap differences: -(9252--9255) -10490 -(10577--10578) -(16585--16589) -295391 -(682164--682165)
Fix<y>? yes
Free blocks count wrong for group #0 (2756, counted=2768).
Fix<y>? yes
Free blocks count wrong for group #9 (2784, counted=2785).
Fix<y>? yes
Free blocks count wrong for group #20 (14702, counted=14704). Fix<y>? yes
Free blocks count wrong (718736, counted=718751).
Fix<y>? yes
Inode bitmap differences: -58591
Fix<y>? yes
Free inodes count wrong for group #7 (6283, counted=6284).
Fix<y>? yes
Directories count wrong for group #7 (1133, counted=1121).
Fix<y>? yes
Free inodes count wrong (322025, counted=322026).
Fix<y>? yes

scm: ***** FILE SYSTEM WAS MODIFIED *****
scm: 71190/393216 files (0.1% non-contiguous), 854113/1572864 blocks fsck from util-linux 2.32.1
e2fsck 1.44.4 (18-Aug-2018)

service vg0 stop
service mdraid stop

ctrl-alt-del

I was able to reboot with init 1 from grub then init 4 from tty1 as root with kernel 4.19.4. Filesystems were clean.

exit from tty1

log in again as normal user under X
su - root
next bisect in action

I must admit, it is time consuming.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #53 from Rainer Fiebig ([email protected]) --- [...]

You have been warned. ;)

But in the end you will be rewarded with something like this:

1234xx56789yy is the first bad commit
...

And honors and glory will rain down on you!
OK, this may be a bit exaggerated. ;)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #54 from Henrique Rodrigues ([email protected]) --- (In reply to Theodore Tso from comment #46)

The bad news is that I've seemed to have made a mistake and there are more changes than that one.

The other bad news is that I got another corruption even with CONFIG_SCSI_MQ_DEFAULT=n.

It's a SATA attached SSD.

I'll attach more information (dmesg, lspci, kernel config, etc). Unfortunately fsck now tells me I've got a bad magic number in super-block, so I think I better start copying some stuff over to another disk before attempting anything else.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #55 from [email protected] ---
I didn't make it with 4.18.0-radeon-07013-g54dbe75bbf-dirty because the radeon module gives me a black screen.

With 4.18.0-radeon-03131-g0a957467c5-dirty, ext4 filesystems were stable but 2hours later an exception followed by a sudden reboot w/o warning. Next try, immediate reboot.
Also bad too.

During bzImage compilation, ld returned:
ld.bfd: arch/x86/boot/compressed/head_64.o: warning: relocation in read-only section `.head.text'
ld.bfd: warning: creating a DT_TEXTREL in object

Is that something suspicious for you?

FIK, I was stuck many times with the following message until I realized usr/.initramfs_data.cpio.xz.d file were not removed from the directory (sig).

# make                                                                 
  CALL    scripts/checksyscalls.sh
  DESCEND  objtool
  CHK     include/generated/compile.h
usr/Makefile:48: *** motifs de cible multiples. Arrêt.
make: *** [Makefile:1041: usr] Error 2

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #56 from Rainer Fiebig ([email protected]) ---
FWIW: I've installed a defconfig-4.19.3 in a VirtualBox-VM. But our bug hasn't shown up so far.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Guenter Roeck ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #57 from Guenter Roeck ([email protected]) ---
I have seen the problem on two of four systems running v4.19.4. All systems are

System 1:
MSI B450 TOMAHAWK (MS-7C02)
Ryzen 2700X
Drive 1: NVME (500GB)
Drive 2: SATA HDD (WD4001FAEX-00MJRA0, 4TB) Problem seen on SATA HDD, with both 4.19.3 and 4.19.4

System 2:
MSI B350M MORTAR (MS-7A37)
Ryzen 1700X
Drive 1: SSD Samsung SSD 840 PRO 250GB
Drive 2: SSD Samsung SSD 840 EVO 250GB
Problem seen on both drives, with both 4.19.3 and 4.19.4

System 3:
Gigabyte AB350M-Gaming 3
Ryzen 1700X
Drive 1: SSD Samsung SSD 840 PRO 250GB
Drive 2: SSD M4-CT256M4SSD2 (250GB)
Problem not seen (yet)

System 4:
MSI B350M MORTAR (MS-7A37)
Ryzen 1700X
Drive 1: NVME (500GB)
Problem not seen (yet)

Default configuration was CONFIG_SCSI_MQ_DEFAULT=y. I tried with CONFIG_SCSI_MQ_DEFAULT=n on system 2 (with 4.19.4) and hit the problem again almost immediately.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #58 from Henrique Rodrigues ([email protected]) --- Created attachment 279685
--> https://bugzilla.kernel.org/attachment.cgi?id=279685&action=edit
debug info

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #59 from Theodore Tso ([email protected]) --- Created attachment 279687
--> https://bugzilla.kernel.org/attachment.cgi?id=279687&action=edit
Ext4 from 4.18

I'm pretty sure the problem is not in the ext4 changes between 4.18 and 4.19, since the changes are all quite innocuous (and if it was in the ext4 code, the regression testing really should have picked it up).

But just to rule things out, I've uploaded the contents of fs/ext4 from 4.18.
I've verified it can be transplanted on top of 4.19 kernel. Could the people
who are experiencing problems with 4.19 try building a kernel with the 4.18
fs/ext4 directory? If you still see problems, then the problem has to be
elsewhere. If you don't, then we can take a closer look at the ext4 changes
(although I'd then be really puzzled why it's only showing up for some folks, but not others).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Theodore Tso ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

 Attachment #279687|Ext4 from 4.18              |Ext4 from 4.18 (tar.gz)
        description|                            |

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #60 from Theodore Tso ([email protected]) ---
Henrique -- what is dm-0? How is it configured? And are you using discard
(either the mount option, or fstrim)? Thanks!!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #61 from Henrique Rodrigues ([email protected]) --- (In reply to Theodore Tso from comment #60)

dm-o is a LUKS encrypted partition that I use as /. I have fstrim running weekly with "fstrim -av" (Ubuntu's default).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #62 from [email protected] ---
If you don't mind, I'll continue bisecting.

But generating a kernel becomes harder with the genuine kernel. Three times a raw I was unable to compile the kernel. It fails with, arch/x86/entry/vdso/vclock_gettime-x32.o:vclock_gettime.c:fonction__vdso_gettimeofday : erreur : débordement de relocalisation : référence à « vvar_vsyscall_gtod_data »

If it fails again, I will need to patch it:

@@ -97,6 +97,7 @@
 
 CPPFLAGS_vdsox32.lds = $(CPPFLAGS_vdso.lds)
 VDSO_LDFLAGS_vdsox32.lds = -Wl,-m,elf32_x86_64 \
+        -fuse-ld=bfd \
         -Wl,-soname=linux-vdso.so.1 \
         -Wl,-z,max-page-size=4096 \
         -Wl,-z,common-page-size=4096

ld gold is my default.

And sorry, I should have patched l1tf_vmx_mitigation too. My mistake.

index 27830880e7a7..cb4a16292aa7 100644
@@ -664,10 +664,9 @@ void x86_spec_ctrl_setup_ap(void)
 enum l1tf_mitigations l1tf_mitigation __ro_after_init = L1TF_MITIGATION_FLUSH;
 #if IS_ENABLED(CONFIG_KVM_INTEL)
 EXPORT_SYMBOL_GPL(l1tf_mitigation);
-
+#endif
 enum vmx_l1d_flush_state l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
 EXPORT_SYMBOL_GPL(l1tf_vmx_mitigation);
-#endif
 
 static void __init l1tf_select_mitigation(void)
 {

Neverless the ext4 issue as it seems to be doesn't make sens. I can compile packages during the test to maintain the cpu's activity on top to ease reproducing the issue.
Each time I do a reboot, I do a fsck on the ext4 partitions (in both rescue mode and normal init process) and it's like for some partitions e2fsck is unable to handle (in any undetermined circumstances) 'Structure needs cleaning' issue (remember my remark about fsck -D). If that's confirmed, a corrupt fs could still be corrupt on the next reboot and misguide us.

In that case, Jens Axboe 4.19.4 patch does its work. I'm bisecting the kernel on a 4.19.4 patched kernel version. The only fs that's stay corrupt after each reboot is my backup partition (sig).
Could someone investigate in that direction please ?

I'm using e2fsprogs 1.44.4 package.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #63 from Michel Dänzer ([email protected]) --- (In reply to Jason Gambrel from comment #3)

FWIW, I was running into ext4 metadata corruption every few days with 4.19 using swap files (on the ext4 / on LVM on LUKS). On a hunch, switched to a swap partition on LVM on LUKS two weeks ago, and haven't run into it since. Swap files were working fine with pre-4.19 kernels. In case it matters, I run fstrim in a weekly cronjob, with discard enabled in /etc/lvm/lvm.conf and /etc/crypttab.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #64 from Jens Axboe ([email protected]) --- (In reply to Jimmy.Jazz from comment #62)

How certain are you that my 4.19 patch fixes the issue completely for you? If 100%, can you also try with 4.19.4 + just the first hunk of that patch? In other words, only apply the part to block/blk-core.c, not the one to block/blk-mq.c

Thanks!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Néstor A. Marchesini ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #65 from Néstor A. Marchesini ([email protected]) --- Hi.
My distro is gentoo testing, I also use 4 partitions raid1 ,are two discs of 1T WD black that have never failed.
These raid1 partitions use mdadm with metadata 0.90.

$ lsblk /dev/md*
RM RO MODEL NAME LABEL      FSTYPE MOUNTPOINT   SIZE PHY-SEC LOG-SEC MODE      
 0  0       md0  GentooBoot ext4                128M     512     512 brw-rw----
 0  0       md1  GentooSwap swap   [SWAP]         4G     512     512 brw-rw----
 0  0       md2  GentooRaiz ext4   /             50G     512     512 brw-rw----
 0  0       md3  GentooHome ext4   /home      877,4G     512     512 brw-rw----

Effectively from 4.19.0 I started to have problems with the boot, the system always closed perfectly unmount all the partitions, but when booting the next time I fall in fsck and end in recovery console, ignoring these errors, restart again and I choose the kernel 4.18.20 and it does not fall in fsck, it also does
not detect any error in the ext4 partitions.
Sometimes these errors trigger the resynchronization of the partition that fsck detects false positives, I see it with $ cat /proc/mdstat
For now I will continue using 4.18.20, the faults I have been doing since 4.19.0
4.19.1 4.19.2 4.19.3 4.19.4 and 4.19.5, given that this is something from the 4.19.x branch

$ uname -a
Linux pc-user 4.18.20-gentoo #1 SMP PREEMPT Sat Nov 24 14:39:41

$ eselect kernel list 
Available kernel symlink targets:
  [1]   linux-4.18.20-gentoo
  [2]   linux-4.19.4-gentoo
  [3]   linux-4.19.5-gentoo *

Regards

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #66 from [email protected] ---
(In reply to Jens Axboe from comment #64)

Without your patch the failure was mostly systematic in the time. Synchronization mechanism is not trivial anyway. But statically there is hope.

I understand, probably syncronize_rcu was a bit too much :). Let me 24h please.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Laurent Bonnaud ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #67 from Laurent Bonnaud ([email protected]) --- I am also experiencing ext4 corruptions with 4.19.x kernels.

One way to trigger this bug that works almost every time on my system is to backup the whole FS with BorgBackup using this command:

nice -19 ionice -c3 borg create -v --stats --list --filter=AME --one-file-system --exclude-caches --compression zstd --progress my-server:/borg-backup::'{hostname}-{now:%Y-%m-%d_%H:%M}' /

Here are kernel messages:

[  916.082499] EXT4-fs error (device sda1): ext4_iget:4831: inode #6318098:
comm borg: bad extra_isize 35466 (inode size 256)
[  916.093908] Aborting journal on device sda1-8.
[  916.096417] EXT4-fs (sda1): Remounting filesystem read-only
[  916.096799] EXT4-fs error (device sda1): ext4_iget:4831: inode #6318101:
comm borg: bad extra_isize 35466 (inode size 256)
[  916.101544] EXT4-fs error (device sda1): ext4_iget:4831: inode #6318103:
comm borg: bad extra_isize 35466 (inode size 256)
[  916.106531] EXT4-fs error (device sda1): ext4_iget:4831: inode #6318107:
comm borg: bad extra_isize 35466 (inode size 256)
[  916.111039] EXT4-fs error (device sda1): ext4_iget:4831: inode #6318110:
comm borg: bad extra_isize 35466 (inode size 256)
[  916.115763] EXT4-fs error (device sda1): ext4_iget:4831: inode #6318112:

comm borg: bad extra_isize 35466 (inode size 256)

If there is some interest, I can provide more details, but in another bug report since this one is already loaded with attached files.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #68 from [email protected] ---

# uname -a
Linux seal 4.18.0-rc1-radeon-00048-ge1333462e3-dirty #36 SMP PREEMPT Wed Nov 28
18:30:01 CET 2018 x86_64 AMD A10-5800K APU with Radeon(tm) HD Graphics
AuthenticAMD GNU/Linux

I finally run a promising kernel that compiles, doesn't crash and cares about my filesystems.
4.18.0-rc1-radeon-00048-ge1333462e3-dirty could be a winner.

This time the backup file system dm-4 could be efficiently cured and dirvish has done its work has expected.
I could compile with it the kernel 4.19.4 as J.Axboe asked me to.

@T.Tso, if you still have an interest in dmesg, fsck (quite impressive) output with that kernel version, let me know.
Actually, the interaction between e2fsck 1.44.4 and kernel 4.18 differs from 4.19

An dmesg excerpt,

[12421.017028] EXT4-fs warning (device dm-4): kmmpd:191: kmmpd being stopped
since filesystem has been remounted as readonly.
[12434.457445] EXT4-fs warning (device dm-4): ext4_multi_mount_protect:325: MMP

interval 42 higher than expected, please wait.
The warning didn't show off with kernel 4.19 and remount is slower. No ext4 errors to see.

# git bisect log
git bisect start
# good: [94710cac0ef4ee177a63b5227664b38c95bbf703] Linux 4.18
git bisect good 94710cac0ef4ee177a63b5227664b38c95bbf703
# bad: [9ff01193a20d391e8dbce4403dd5ef87c7eaaca6] Linux 4.20-rc3
git bisect bad 9ff01193a20d391e8dbce4403dd5ef87c7eaaca6
# bad: [9ff01193a20d391e8dbce4403dd5ef87c7eaaca6] Linux 4.20-rc3
git bisect bad 9ff01193a20d391e8dbce4403dd5ef87c7eaaca6
# bad: [84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d] Linux 4.19
git bisect bad 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d
# bad: [f48097d294d6f76a38bf1a1cb579aa99ede44297] dt-bindings: display:
renesas: du: Document r8a77990 bindings
git bisect bad f48097d294d6f76a38bf1a1cb579aa99ede44297
# bad: [f48097d294d6f76a38bf1a1cb579aa99ede44297] dt-bindings: display:
renesas: du: Document r8a77990 bindings
git bisect bad f48097d294d6f76a38bf1a1cb579aa99ede44297
# bad: [54dbe75bbf1e189982516de179147208e90b5e45] Merge tag
'drm-next-2018-08-15' of git://anongit.freedesktop.org/drm/drm
git bisect bad 54dbe75bbf1e189982516de179147208e90b5e45
# bad: [0a957467c5fd46142bc9c52758ffc552d4c5e2f7] x86: i8259: Add missing
include file
git bisect bad 0a957467c5fd46142bc9c52758ffc552d4c5e2f7
# bad: [958f338e96f874a0d29442396d6adf9c1e17aa2d] Merge branch 'l1tf-final' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 958f338e96f874a0d29442396d6adf9c1e17aa2d
# bad: [85a0b791bc17f7a49280b33e2905d109c062a47b] Merge branch 'for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
git bisect bad 85a0b791bc17f7a49280b33e2905d109c062a47b
# bad: [8603596a327c978534f5c45db135e6c36b4b1425] Merge branch
'perf-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 8603596a327c978534f5c45db135e6c36b4b1425
# bad: [2406fb8d94fb17fee3ace0c09427c08825eacb16] Merge branch
'sched-urgent-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 2406fb8d94fb17fee3ace0c09427c08825eacb16
# bad: [cd23ac8ddb7be993f88bee893b89a8b4971c3651] rcu: Add comment to the last
sleep in the rcu tasks loop
git bisect bad cd23ac8ddb7be993f88bee893b89a8b4971c3651

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #69 from Theodore Tso ([email protected]) ---
Hi Jimmy, how certain are you that e1333462e3 is stable for you? i.e., how
long have you been running with that kernel and how quickly do your other git bisect bad build fail for you?

And I assume you have run a forced fsck (ideally while 4.18 is booted) on the file system before installing each kernel that you were bisect testing, right?
Otherwise it's possible that a previous bad kernel had left the file system
corrupted, and so a particular kernel stumbled on a corruption, but it wasn't actually *caused* by that kernel.

The reason why I'm asking these question is that based on your bisect, it would *appear* that the problem was introduced by an RCU change. If you look at the output of "git log --oneline e1333462e3..cd23ac8ddb7" all of the changes are
RCU related. That's a bit surprising, since given that only some users are
seeing this problem. If there was a regression was introduced in the RCU subsystem, I would have expected a large number of people would have been complaining, with many more bugs than just in ext4.

And there is some evidence that your file system has gotten corrupted. The warnings you report here:

[12421.017028] EXT4-fs warning (device dm-4): kmmpd:191: kmmpd being stopped
since filesystem has been remounted as readonly.
[12434.457445] EXT4-fs warning (device dm-4): ext4_multi_mount_protect:325: MMP 

interval 42 higher than expected, please wait.

Are caused by the MMP feature being enabled on your kernel. It's not enabled by default, and unless you have relatively exotic hardware (e.g., dual-attached SCSI disks that can be reached by two servers for failover) there is no reason
to turn on the MMP feature. You can disable it via: "tune2fs -O ^mmp
/dev/dm-4". (And you can enable it via "tune2fs -O mmp /dev/dm-4".) So
apparently while you were running your tests, the superblock had at least one bit (the MMP feature bit) flipped by a rogue kernel.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #70 from Néstor A. Marchesini ([email protected]) --- (In reply to Theodore Tso from comment #59)

I copied /fs/ext4 from tree 4.18.20 to tree 4.19.5
and compile everything from scratch the tree 4.19.5.
Well, now we'll have to wait and cross our fingers every time I restart the PC. So far I had no problems, if they appear I would be posted again with data. Regarding my configuration of CONFIG_SCSI_MQ_DEFAULT it was always enabled for eons.

# cat /boot/config-4.18.20-gentoo |grep CONFIG_SCSI_MQ_DEFAULT=
CONFIG_SCSI_MQ_DEFAULT=y
# cat /boot/config-4.19.4-gentoo |grep CONFIG_SCSI_MQ_DEFAULT=
CONFIG_SCSI_MQ_DEFAULT=y
# cat /boot/config-4.19.5-gentoo |grep CONFIG_SCSI_MQ_DEFAULT=
CONFIG_SCSI_MQ_DEFAULT=y

# eix -Ic e2fsprogs
[I] sys-fs/e2fsprogs (1.44.4@07/11/18): Standard EXT2/EXT3/EXT4 filesystem
utilities
[I] sys-libs/e2fsprogs-libs (1.44.4@06/11/18): e2fsprogs libraries (common
error and subsystem)
Found 2 matches

Regards

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Ortwin Glück ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #71 from Ortwin Glück ([email protected]) ---
If it helps, I do NOT see this bug and I've run all 4.18.y and 4.19.y kernels:
CONFIG_SCSI_MQ_DEFAULT=y
CONFIG_MQ_IOSCHED_DEADLINE=y

rootfs on RAID-0 on 2 SSDs:
cat /proc/mdstat
Personalities : [raid0]
md127 : active raid0 sdb1[1] sda3[0]

      499341824 blocks super 1.2 256k chunks

/dev/md127 on / type ext4 (rw,noatime,discard,stripe=128)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #72 from [email protected] ---
(In reply to Theodore Tso from comment #69)
I didn't trust the kernel enough to let it work all the night without close observation (i.e I need some rest).

In comparison with the latest tests, I feel certain the kernel is good after one day with parallel running compilations.That's why I postponed J.Axboe request.

Actually, I'm working with 4.18 e1333462e3 and after three clean reboot, the disks stayed clean.

Dirvish is running today and nothing bad has append. I can say 4.18 e1333462e3 is good.

$ uptime
 17:12:44 up  3:23,  6 users,  load average: 10,54, 10,99, 10,13

Also, I didn't change my .config except when asked during the current commit.

The builds failed after I solicit the kernel or when I back up the system (dirvish/rsync). When the activity is low I didn't observe anything suspicious.

Also, the server is not a stupid idle beagle.

To resume,

- I jumped to 4.19 because they were no improvement with 4.20-c3... and I feared for my datas.
- From f48097d2 to 54dbe75b radeon module didn't work (i.e no display)
- 0a957467c5 crashed. Next try, crashed immediately during the boot. (comment
55)
- 958f338e I missed 'l1tf' patch (comment 62)
- From 958f338e to cd23ac8d I missed 'vdso' patch (comment 62)
- e1333462e3 I applied both patches 'l1tf' and 'vdso'

With commit e1333462e3, dm-4 partition could be cleaned efficiently (see attachement).

I have run a fsck /dev/dm-XX with 4.18 commit e1333462e3 first in rescue mode than from init script during normal boot. It was not necessary to force an fsck distinguished from 4.19 and higher releases.

I thought about it too (comment 62 second paragraph). In that case, why does only 4.18 + e2fsprogs be able to clean the partitions and not with more recent kernels ? Doesn't e2fsprogs be compatible with 4.19 branch, does it ?

I'm using gcc (Gentoo 8.2.0-r4 p1.5) 8.2.0 and use LD=ld.bfd. My linker is gold by default. Sadly, I didn't find a way to compile it with clang.

I understand. But race conditions are not always trivial.

dm-4 is marked read only until a backup is performed. I add (temporarily) mmp to the file systems because I though I had a multi remount issue at first. The report what intended to attract your attention on the following; remount,rw or remount,ro are really slow with 4.18 commit e1333462e3 and the warning has never appeared in that way on other builds. That was not observed with vanilla 4.18.X.

Please, I didn't intend to misguide you. Just consider the warning as a false positive. If the warning show of a rogue kernel, then it is the kernel 4.18 (a contradiction).

My computers are on ups and I do an fsck on every reboot but force it again only when an error has been detected. Anyway, corruptions that appear and disappear all of sudden on the majority of fs with such a frequency is quite remarkable.

The file systems are now clean over reboots. I propose to test if 4.19.5 kernel stops showing corruptions. If they stop, it still opens a new question, why was fsck missing some file system corruptions ?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #73 from [email protected] ---
Created attachment 279739
--> https://bugzilla.kernel.org/attachment.cgi?id=279739&action=edit
fsck output kernel 4.18

the fsck has been done in rescue mode w/ 4.18

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Michael Orlitzky ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #74 from Michael Orlitzky ([email protected]) --- (In reply to Laurent Bonnaud from comment #67)

Ouch, me too. I've already been through two hard drives and a new SATA controller. I was just about to resign myself to replacing the whole PC.

My system is an older AMD Phenom, with absolutely nothing fancy going on. Boring spinning disks, no RAID, and exactly the symptom above.

After upgrading to 4.19.0 everything was fine for a week, and then Borg started reporting these errors. If I boot to a rescue CD and fsck, things go back to "normal," but then after a few more days I get corruption again. IIRC I skipped 4.19.1 but had the same problem with 4.19.2, and now again on 4.19.3.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #75 from [email protected] ---
(In reply to Jens Axboe from comment #64)

@T.Tso and J.Axboe
e1333462e3 was not able to compile the 4.19.5 kernel.

Long story, gcc begins to complain of missing elfutils package (it was installed already). I felt also in an old CONFIG_UNWINDER_ORC bug "Cannot generate ORC metadata". Compilations begin to fail with a "cannot make executable" error. As unbelievable it is, the bug was reported recently
(https://lkml.org/lkml/2018/11/5/108).
I'm using dev-libs/elfutils-0.175 and the kernel isn't affected by
https://bugs.gentoo.org/671760

The good news. 4.19.4 kernel with the part to 'block/blk-core.c' of your patch has compiled 4.19.5. It doesn't show any sign of ext4 corruption.

I'm waiting for the next backup tomorrow.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #76 from [email protected] ---
(In reply to Jimmy.Jazz from comment #75)

@J.Axboe
No need to wait. ext4 error resurfaced on dm-8 this time. block/blk-core.c patch doesn't correct the issue.

[ 3774.584797] EXT4-fs error (device dm-8): ext4_iget:4985: inode #1614666:

comm emerge: bad extended attribute block 1

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #77 from Jens Axboe ([email protected]) --- (In reply to Jimmy.Jazz from comment #76)

Are you still confident the full patch works? It's interesting since that has RCU, and the other changes point in that direction, too.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #78 from Néstor A. Marchesini ([email protected]) --- Well, guys, this seems to pass the test, after several reboots every several hours of use, I had no more corruptions of my four partitions mdadm raid1. The solution was to delete the ext4 folder from my kernel 4.19.5 and copy the ext4 folder from my previous kernel 4.18.20 and recompiling the tree 4.19.5.

$ uname -a
Linux pc-user 4.19.5-gentoo #1 SMP PREEMPT Thu Nov 29 00:45:31 2018 x86_64 AMD
FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux

I was comparing both folders /fs/ext4 of the 4.18.20 and 4.19.5 trees with meld and there are several modifications, unfortunately it exceeds my knowledge. At least on this side I affirm that the problem is gone, we will see it happen with later patches.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #79 from Hao Wei Tee ([email protected]) --- (In reply to Néstor A. Marchesini from comment #78)

If this is the case, perhaps you could bisect fs/ext4 between tags v4.18 and v4.19?

$ git bisect start v4.19 v4.18 -- fs/ext4

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #80 from Rainer Fiebig ([email protected]) --- (In reply to Néstor A. Marchesini from comment #78)

If you can bisect it as suggested in comment 79, please mind what Ted Tso has said in comment 69, para. 2.

So, after you have hit a bad kernel, make sure that your fs is OK and do the next step (compiling) with a known-as-good-kernel (4.18.20). Otherwise you might get false negatives (wrong bads).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #81 from Rainer Fiebig ([email protected]) --- (In reply to Jens Axboe from comment #77)

It looks like the problem may be caused by changes in fs/ext4 (see comment 78).

But I'm wondering why this only affects some (quite a few, though) and not all. Like others, I'm running 4.19.5 without any problem here, it's just nice.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #82 from Hao Wei Tee ([email protected]) ---
I fear the bug might be caused by some interaction between something new in fs/ext4 and something new elsewhere... Sounds unlikely, but it's possible.

Since 4.18 ext4 seems to work on 4.19 kernel, maybe it's worth trying 4.19 ext4 on 4.18 kernel (before a bisect), just to make sure the bisect won't lead us down a false trail?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #83 from Rainer Fiebig ([email protected]) --- (In reply to Hao Wei Tee from comment #82)

Interesting idea. But what works in one direction might not necessarily work the other way round. Personally, I'd rather like to be on the safe side here. So before doing this it might be wise to here what Ted Tso thinks about it, just IMO.

And I don't think that bisecting just fs/ext4 would be misleading. If we find a bad commit there, Ted and others will look at it anyway and will see whether this alone explains the problems or whether an interaction with something else would be necessary to make sense of it.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Andreas John ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #84 from Andreas John ([email protected]) ---
Hi,
thanks for investigating the issue. I "costed" my some inodes on my ext4 rootfs , rMBP, SSD, dm-crypt disk. It appeared on 4.19.1 my case.

I just wanted to add that I run btrfs / dmcrypt /samessd on the /home and that one is not affected by that issue as far as I can tell.

rgds,
j

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #85 from Hao Wei Tee ([email protected]) --- (In reply to Rainer Fiebig from comment #83)

Exactly my point. We know that (4.19 ext4 and kernel is broken), (4.18 ext4 and kernel is working), and (4.18 ext4 and 4.19 kernel is working).

If (4.19 ext4 and 4.18 kernel) is broken, then _most likely_ the bug is caused by something that changed in v4.18..v4.19. If (4.19 ext4 and 4.18 kernel) *works*, then either the bug is in something else that changed, or there is an interaction between two changes that happened in v4.18..v4.19.

In any case, bisecting v4.18..v4.19 will probably give us a clue.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #86 from Rainer Fiebig ([email protected]) --- (In reply to Hao Wei Tee from comment #85)

Sure, I understand this. I would just shy away from recommending this to others without a nod from higher powers. But of course it's up to Nestor whether he wants to try this or not.

Let's hope for the best. Perhaps bisecting fs/ext4 will provide enough of a clue already and spare the poor bisecter to have to bisect the whole beast. ;)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #87 from Artem S. Tashkinov ([email protected]) --- Regression testing could be carried out in a VM running on top of a ramdisk (e.g. tmpfs) to speed up the process.

I guess someone with a decent amount of persistence and spare time could do that and test each individual commit between 4.18 and 4.19, however that doesn't guarantee success since the bug might be hardware related and not reproducible in a virtual environment. Or it might require obscene amounts of RAM/disk space which would be difficult, if not impossible to reproduce in a VM.

I for one decided to stay on 4.18.x and not upgrade to any more recent kernels until the regression is identified and dealt with.

Maybe one day someone will become truly invested in the kernel development process and we'll have proper QA/QC/unit testing/regression testing/fuzzying, so that individuals won't have to sacrifice their data and time because kernel developers are mostly busy with adding new features and usually not really concerned with performance, security and stability of their code unless they are pointed at such issues.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #88 from [email protected] ---
(In reply to Jens Axboe from comment #77)

4.19.4 full patched is stable. I'm just puzzled in its capability to clean a failed file system with sys-fs/e2fsprogs-1.44.4.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #89 from [email protected] ---
To all,

Please add a large among of read-write mounted file systems attached to your test system. That will increase the probability of the failure. My experience is, the issue doesn't affect a specific mountpoint over and over but rather a random one.

FYI, I didn't have any issue with one of the tmpfs filesystems installed. You should take it into consideration when creating your VM test environment.

nilfs is stable.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #90 from Hao Wei Tee ([email protected]) --- (In reply to Artem S. Tashkinov from comment #87)

What we have now is not proper? syzkaller bot, Linux Test Project, kernelci.org, xfstests, and more that I don't know of. Probably more than any other OS.

I think it's fair to say Linux has by far more configuration options than any other kernel out there. It's not feasible to test every single possible combination. Things will slip through the cracks, especially bugs like this one where clearly there is something wrong, but not everyone is able to reproduce it at all. Automated tests are going to miss bugs like this.

We're not doing things worse than anyone else. Apple's APFS had major issues. Microsoft just had a big problem with their Windows 10 1809 rollout.

Anyway, I remember you from another post you made to LKML complaining about Linux. You really don't like the way Linux is developed. Why do you still use it?

I digress.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #91 from Néstor A. Marchesini ([email protected]) --- For me, the problem started with the release of 4.19.0, and looking at the commits of the tree 4.19.0, I see that many things of ext4 have been changed ... very many I would say.
If you search with ext4 within the list of comnits you will find several and with very important changes.

https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.19

There are several massive, one of the most important is:

https://github.com/torvalds/linux/commit/c140f8b072d16595c83d4d16a05693e72d9b1973

This weekend I will try with git bisect, but it will be a very time-consuming task due to the large number of ext4 commits.
I'm still using 4.19.5 with the ext4 folder of 4.18.20. I have not had problems so far.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #92 from Hao Wei Tee ([email protected]) --- (In reply to Néstor A. Marchesini from comment #91)

There are only 32 new commits in fs/ext4 in v4.19 from v4.18. See [1], count until commit "ext4: fix check to prevent initializing reserved inodes".

[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/fs/ext4?id=v4.19

This isn't in v4.19? It only got pulled in the v4.20 merge window.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #93 from Guenter Roeck ([email protected]) ---
Most of the ext4 patches in v4.19 have been backported to v4.18.y. Since v4.18.20 is reported to be stable, it is quite likely that the problem lies with one or more of the patches which have _not_ been backported. This would be one of the following patches.

ext4: close race between direct IO and ext4_break_layouts()
ext4: add nonstring annotations to ext4.h
ext4: readpages() should submit IO as read-ahead
dax: remove VM_MIXEDMAP for fsdax and device dax
ext4: remove unneeded variable "err" in ext4_mb_release_inode_pa()
ext4: improve code readability in ext4_iget()
ext4: handle layout changes to pinned DAX mappings
ext4: use swap macro in mext_page_double_lock
ext4: check allocation failure when duplicating "data" in ext4_remount()
ext4: fix warning message in ext4_enable_quotas()
ext4: super: extend timestamps to 40 bits
ext4: use timespec64 for all inode times
ext4: use ktime_get_real_seconds for i_dtime
ext4: use 64-bit timestamps for mmp_time
block: Define and use STAT_READ and STAT_WRITE

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #94 from Jens Axboe ([email protected]) ---
In terms of ext4 changes, it'd be interesting to just revert this one:

commit ac22b46a0b65dbeccbf4d458db95687e825bde90

Author: Jens Axboe <[email protected]>
Date:   Fri Aug 17 15:45:42 2018 -0700

ext4: readpages() should submit IO as read-ahead

as that guy is generally just not trust worthy. In all seriousness, though, it shouldn't cause issues (or I would not have done it), and we already do this for readpages in general, but I guess we could have an older bug in ext4 that depends deeply on read-ahead NOT failing. Not sure how likely that is, Ted can probably comment on that.

But it's a trivial revert, and it could potentially be implicated.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #95 from Jens Axboe ([email protected]) ---
BTW, if that patch is to blame, then the bug is elsewhere in ext4 as there should be no way that read-ahead failing should cause corruption.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #96 from Rainer Fiebig ([email protected]) --- (In reply to Jens Axboe from comment #94)

Jens, could you provide the patch here, so that perhaps Jimmy and Nestor can revert it on their 4.19.x and tell us what they see?
Thanks.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #97 from Guenter Roeck ([email protected]) ---
#94 makes me wonder if the problem may be related to
https://lkml.org/lkml/2018/5/21/71. Just wondering, and I may be completely off track, but that problem is still seen against the mainline kernel.

#96: commit ac22b46a0b65 can be reverted cleanly with "git revert".

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #98 from Rainer Fiebig ([email protected]) --- (In reply to Guenter Roeck from comment #97)

Yep, but perhaps some people here don't use git and/or have cloned the repo.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #99 from Guenter Roeck ([email protected]) --- #98: Good point.

I am going to give it a try with the following on top of v4.19.5:

Revert "ext4: handle layout changes to pinned DAX mappings" Revert "dax: remove VM_MIXEDMAP for fsdax and device dax" Revert "ext4: close race between direct IO and ext4_break_layouts()" Revert "ext4: improve code readability in ext4_iget()" Revert "ext4: readpages() should submit IO as read-ahead"

Wild shot, but I figured it may be worth try.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #100 from Guenter Roeck ([email protected]) ---
So far I have been unable to reproduce the problem after reverting the patches mentioned in #99. I'll now install this kernel on a second previously affected system. I'll report back tomorrow morning.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #101 from Theodore Tso ([email protected]) ---
Guenter, what is your kernel config? A number of these changes are related to
CONFIG_DAX. Are you building kernels with or without CONFIG_DAX enabled?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #102 from Guenter Roeck ([email protected]) --- Enabled:

$ grep DAX .config
CONFIG_NVDIMM_DAX=y
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_FS_DAX=y
CONFIG_FS_DAX_PMD=y

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #103 from Guenter Roeck ([email protected]) ---
It doesn't look like dax is loaded, though. /dev/daxX does not exist on any of the affected systems, and lsmod doesn't show any dax modules.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #104 from Rainer Fiebig ([email protected]) --- #101 Both 4.19.x kernels (VM/real HW):

# CONFIG_DAX is not set
# CONFIG_FS_DAX is not set

Both kernels did *not* have the problem.

This may explain why some see the problem and others don't.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #105 from Andreas John ([email protected]) ---
FYI: 4.19.6 was just released, the is a DAX fixup inside:

dax: Avoid losing wakeup in dax_lock_mapping_entry

Maybe the courageous testers should also consider that one, if the issue is DAX related?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #106 from Guenter Roeck ([email protected]) ---
I have seen the problem again tonight, but I am not sure if I cleaned the affected file system correctly with an older kernel before I started the test. I'll keep running with the same reverts for another day.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #107 from Guenter Roeck ([email protected]) --- #104: Possibly, but it doesn't explain why I see the problem only on two of four systems, all running the same kernel.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #108 from Rainer Fiebig ([email protected]) --- (In reply to Guenter Roeck from comment #107)

Right. Bye bye DAX-theory.

I'm wondering by now whether I made a config-mistake somewhere and *that's* why I don't have the problem. ;)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Marc Koschewski ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #109 from Marc Koschewski ([email protected]) --- (In reply to Guenter Roeck from comment #107)

Could it be hardware related like ie. blacklisted "trim" for ie. Samsung 850 Pro? Are the 4 machines absoutely equal hardware-wise (at least on the block layer)? Maybe such a quirk is needed for just another device...

Running ext4 on 4.19.{5,4,3,2,1,0} with not one error with the following setup:

root@marc:~ # lsblk /dev/sda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda              8:0    0  477G  0 disk  
├─sda1           8:1    0    1G  0 part  /boot
├─sda2           8:2    0    2M  0 part  
├─sda3           8:3    0    2M  0 part  
├─sda4           8:4    0    2M  0 part  
├─sda5           8:5    0    1G  0 part  
├─sda6           8:6    0  408G  0 part  
│ └─crypt-home 254:1    0  408G  0 crypt /home
├─sda7           8:7    0   59G  0 part  /
└─sda8           8:8    0    8G  0 part  
  └─crypt-swap 254:0    0    8G  0 crypt 

root@marc:~ # mount | grep home /dev/mapper/crypt-home on /home type ext4 (rw,nosuid,noatime,nodiratime,quota,usrquota,grpquota)

root@marc:~ # cryptsetup status crypt-home
/dev/mapper/crypt-home is active and is in use.
type: LUKS1
cipher: aes-xts-plain64
keysize: 512 bits
key location: dm-crypt
device: /dev/sda6
sector size: 512

  offset:  4096 sectors
  size:    855633920 sectors
  mode:    read/write

root@marc:~ # egrep -i "(ext4|dax)" /boot/config-4.19.5loc64 CONFIG_DAX=y
# CONFIG_DEV_DAX is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT2=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_ENCRYPTION is not set
# CONFIG_EXT4_DEBUG is not set
CONFIG_FS_DAX=y

root@marc:~ # parted --list
Modell: ATA Samsung SSD 860 (scsi)
Festplatte /dev/sda: 512GB
Sektorgröße (logisch/physisch): 512B/512B
Partitionstabelle: gpt
Disk-Flags:

Nummer  Anfang  Ende    Größe   Dateisystem  Name  Flags
 1      1049kB  1075MB  1074MB  ext4
 2      1075MB  1077MB  2097kB                     boot, esp
 3      1077MB  1079MB  2097kB
 4      1079MB  1081MB  2097kB
 5      1081MB  2155MB  1074MB  ext4
 6      2155MB  440GB   438GB
 7      440GB   504GB   63,4GB  ext4
 8      504GB   512GB   8518MB

...
...
...
Modell: Linux device-mapper (crypt) (dm)
Festplatte /dev/mapper/crypt-home: 438GB
Sektorgröße (logisch/physisch): 512B/512B
Partitionstabelle: loop
Disk-Flags:

Nummer  Anfang  Ende   Größe  Dateisystem  Flags
 1      0,00B   438GB  438GB  ext4

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #110 from Bart Van Assche ([email protected]) --- (In reply to Marc Koschewski from comment #109)

Marc, are you using an I/O scheduler? I'm not using an I/O scheduler:

$ cat /sys/block/sda/queue/scheduler 
[none]

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #111 from Marc Koschewski ([email protected]) --- (In reply to Bart Van Assche from comment #110)

I do:

root@marc:~ # cat /sys/block/sda/queue/scheduler
[mq-deadline] kyber bfq none

might be relevant as well:

root@marc:~ # cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.19.5loc64 root=/dev/sda7 ro init=/sbin/openrc-init root=PARTUUID=6d19e60a-72a8-ee44-89f4-cc6f85a9436c real_root=/dev/sda7 ro resume=PARTUUID=fbc25a25-2d09-634d-9e8b-67308f2feddf real_resume=/dev/sda8 acpi_osi=Linux libata.dma=3 libata.noacpi=0 threadirqs rootfstype=ext4 acpi_sleep=s3_bios,s3_beep devtmpfs.mount=0 net.ifnames=0 vmalloc=512M noautogroup elevator=deadline libata.force=noncq nouveau.noaccel=0 nouveau.nofbaccel=1 nouveau.modeset=1 nouveau.runpm=0 nmi_watchdog=0 i915.modeset=0 cgroup_disable=memory scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y vgacon.scrollback_persistent=1 processor.ignore_ppc=1 intel_iommu=off crashkernel=128M apparmor=1 security=apparmor

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Eric Benoit ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #126 from Eric Benoit ([email protected]) ---
FWIW I believe this issue is affecting ZFS as well. I'm getting the occasional checksum error on a random drive of a RAID-Z configuration (five 4T WD Reds).

I'd initially suspected a chipset (Intel 5400) issue as it's spread more or less evenly across the devices. However it's definitely a software issue, as it only occurs running kernel 4.19.[26] and disappears entirely with 4.18.20.

On a different workstation with ZFS and Seagate drives I haven't been able to reproduce the issue.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #127 from Theodore Tso ([email protected]) --- In reply to #118, from James Courtier-Dutton:

While there have been a few people who have reported problems with the contents of their files, the vast majority of people are reporting problems that seem to include complete garbage being written into metadata blocks --- i.e., completely garbage in to inode table, block group descriptor, and superblocks.
This is getting detected by the kernel noticing corruption, or by e2fsck
running and noticing that the file system metadata is inconsistent. More
modern ext4 file systems have metadata checksum turned on, but the reports from e2fsck seem to indicate that complete garbage (or, more likely, data meant for block XXX is getting written to block YYY); as such, the corruption is not subtle, so generally the kernel doesn't need checksums to figure out that the metadata blocks are nonsensical.

It should be noted that ext4 has very strong checks to prevent this from happening. In particular, when a inode's logical block number is converted to a physical block number, there is block_validity checking to make sure that the physical block number for a data block does not map onto a metadata block. This prevents a corrupted extent tree from causing ext4 to try to write data meant for a data block on top of an inode table block, which would cause the sort of symptoms that some users have reported.

One possible cause is that something below ext4 (e.g. the block layer, or an I/O scheduler) is scrambling the block number so that a file write meant for
data block XXX is getting writen to metadata block YYY. If Eric Benoit's
report in comment #126 is to believed, and he is seeing the same behavior with ZFS, then that might be consistent with a bug in the block layer.

However, some people who have reported that transplanting ext4 from 4.18 onto 4.19 has caused the problem to go away. That would be an argument in favor of the problem being in ext4.

Of course, both observations might be flawed (see my previous comments about false positive and negative reports). And there might be more than one bug that we are chasing at the moment.

But the big question which we don't understand is why are some people seeing
it, but not others. There are a huge number of variables, from kernel
configs, to what I/O scheduler might be selected, etc. The bug also seems to
be very flaky, and there is some hint that heavy I/O load is required to trigger the bug. So it might be that people who think their kernel is fine, might actually be buggy, because they simply haven't pushed their system hard enough. Or it might require heavy workloads of a specific type (e.g., Direct I/O or Async I/O), or one kind of workload racing with another type of
workload. This is what makes tracking down this kind of bug really hard.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #128 from Theodore Tso ([email protected]) --- To Guenter, re: #119.

This is just my intuition, but this doesn't "smell" like a problem with a bad drive. There are too many reports where people have said that they don't see
the problem with 4.18, but they do see it with 4.19.0 and newer kernels. The
reports have been with different hardware, from HDD's to SSD's, with some people reporting NVMe-attached SSD And some reporting SATA-attached SSD's.

Can hard drives and SSDs nowadays fail silently by reading bad data instead of
reporting read (and/or write) errors? One of the things I've learned in my
decades of storage experience is to never rule anything out --- hardware will do some very strange things. That being said.... no, normally this would be
highly unlikely. Hard Drive and SSD's have strong error-correcting codes,
parity and super-parity checks in their internal data paths, so silent read errors are unlikely, unless the firmware is *seriously* screwed up.

In addition, the fact that some of the reports involve complete garbage getting written into the inode table, it seems more likely the problem is on the writing side rather than on the read side.

One thing I would recommend is "tune2fs -e remount-ro /dev/sdXX". This will set the default mode to remount the file system read-only. So if the problem is on the read side, it makes it even more unlikely that the data will be
written back to disk. Some people may prefer "tune2fs -e panic /dev/sdXX",
especially on production servers. That way, when the kernel discovers a file
system inconsistency, it will immediately force a reboot, and then the fsck run on reboot can fix up the problem. More importantly, by preventing the system from continuing to operate after a problem has been noticed, it avoids the problem metastasizing, making things even worse.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #112 from Néstor A. Marchesini ([email protected]) ---

$ zcat /proc/config.gz |grep DAX
CONFIG_DAX=m
# CONFIG_FS_DAX is not set

DAX I have it as a module, but I've never seen it loaded with lsmod.

$ cat /sys/block/sda/queue/scheduler
[none]

Always use the mounting parameters in my partitions barrier=1,data=ordered

$ cat /etc/fstab |grep LABEL
LABEL=GentooBoot  /boot    ext4   noatime,noauto,barrier=1,data=ordered    0 2
LABEL=GentooSwap  none     swap   swap  0 0
LABEL=GentooRaiz  /        ext4   noatime,barrier=1,data=ordered   0 1
LABEL=GentooHome  /home    ext4   noatime,barrier=1,data=ordered   0 2

Excellent point given by Guenter Roeck in comment 93
I would have to try to create several trees 4.19.5 and be removed one or two at a time,
to isolate the fault.
I still use 4.19.5 with the ext4 folder of 4.18.20 and zero problems.

Regards

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #113 from Rainer Fiebig ([email protected]) --- (In reply to Néstor A. Marchesini from comment #112)

Have you given up on your plan to bisect this like suggested in comment 79?

It would be only 5 steps for those 32 commits. And compile times should be rather short.

If you know how to reproduce/provoke the errors it could be done within 2 hours or less.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #114 from Theodore Tso ([email protected]) ---
So I've gotten a query off-line about whether I'm still paying attention to
this bug. The answer is that I'm absolutely paying attention. The reason
why I haven't commented much is because there's not much else to say, and I'm still waiting for more information.

On that front --- I am *absolutely* grateful for people who are still trying to debug this issue, especially when it may be coming at the risk of their data. However, one of the challenges is that it's very easy for reports to be either false positives or false negatives.

False positives come from booting a kernel which might be fine, but the file
system was corrupted from running a previous kernel. Remember, when you get
an EXT4-fs error report, that's when the kernel discovers the file system corruption; it doesn't necessarily mean that the currently running kernel is
buggy. To prevent this false positives, please run "e2fsck -fy /dev/sdXX >
/tmp/log.1" to make sure the file system is clear before rebooting into the new kernel. If e2fsck -fy shows any problems that are fixed, please then run "echo 3 > /proc/sys/vm/drop_caches ; e2fsck -fn /dev/sdXX > /tmp/log.2" to make sure the file system is really clean.

False negatives come from booting a kernel which is buggy, but since this bug seems to be a bit flakey, you're getting lucky/unlucky enough to such that after N hours/days, you just haven't tripped over the bug --- or you *have* tripped over the bug, but the kernel hasn't noticed the problem yet, and so it
hasn't reported the EXT4-fs error yet. There's not a lot we can do to
absolutely avoid all false negatives, but if you are running a kernel which you report is OK, and then a day later, it turns out you see corruption, please don't forget to submit a comment to bugzilla, saying, "my comment in #NNN, where I said a particular kernel was problem-free; turns out I have seen a problem with it."

Again, my thanks for trying to figure out what's going on. Rest assure that
Jens Axboe and I are both paying very close attention. This bug is a really
scary one, both because of how the severity of its consequences, *and* because neither of us can reproduce it on our own systems or regression tests --- so we
are utterly reliant on those people who *can* reproduce the issue to give us
data. We very much want to make sure this gets fixed ASAP!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #129 from [email protected] --- (In reply to carlphilippreh from comment #29)

While I wasn't able to reproduce the bug for quite some time, it ended up coming back. I'm currently running 4.19.6 and I see invalid metadata in files that I have written using this version.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #130 from [email protected] ---
Just as I was writing this, my third computer running Linux (currently at 4.19.6) is now also running into this issue.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #131 from Reindl Harald ([email protected]) --- has anybody ever seen that bug within a virtual machine?

i currently run 4.19.x only inside VMs on VMware Workstation / VMware ESXi and did not see any issues, my only phyiscal test was my homeserver which completley crahsed 4 times like because of the VMware Workstation 14 kernel-modules lasted only for a weekend (RAID10, 2 Samsung Evo 850 2 TB, 2 Samsung Evo 860 2 TB) after the last crash left a 0 byte "firewall.sh" in the nested VM i was working for hours

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #132 from Rainer Fiebig ([email protected]) --- (In reply to carlphilippreh from comment #129)

I think it could be helpful if you provided your .config(s) here.

And what kernel are you using: self-compiled/from your distribution (which)? If self-compiled: have you made changes to the .config?

The set-up of the boxes you mentioned in comment 5 seems just right to hunt this down. ;)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #115 from Guenter Roeck ([email protected]) --- Still playing. I have now seen the problem several times with the patches per #99 reverted, I am just not 100% sure if I see false positives.

For those claiming that upstream developers don't care: I for my part do plan to spend as much time on this as needed to nail down the problem, though I have to admit that the comment in #87 almost made me quit (and wonder why I spend time, energy, and money running kerneltests.org).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #116 from Michael Orlitzky ([email protected]) --- Here are two other data points, just for the record:

1. Like comment #65, I've only actually seen this corruption on three
physical
disks, and all of them were Western Digital Caviar Blacks. There is
another
disk in my system -- a different model -- that has been lucky so far; but this may be pure chance. My /dev/sda has had multiple problems, but /dev/sdb only got corrupted once even though they're the same model.

2. The corruption for me is occurring in files (contained in directories)
that
I haven't touched in a long time. They get backed up -- which means that they get read -- but few if any have been written recently. In addition, all of my mounts are "noatime." Normally I wouldn't expect corruption from *reading* files, which is what lead me to start swapping out disks and SATA controllers.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #133 from [email protected] ---
Created attachment 279779
--> https://bugzilla.kernel.org/attachment.cgi?id=279779&action=edit
Config of first computer

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #134 from [email protected] ---
Created attachment 279781
--> https://bugzilla.kernel.org/attachment.cgi?id=279781&action=edit
Config of second computer

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #117 from Jens Axboe ([email protected]) --- (In reply to Artem S. Tashkinov from comment #87)

You obviously have no idea wtf you are talking about, I suggest you go investigate just how much testing is done, continuously, on things like file system and storage. I take personal offense in in claims that developers "are not really concerned with performance and stability of their code".

Here's a news flash for you - bugs happen, no matter how much testing is done on something.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #135 from [email protected] ---
Doesn't the Linux kernel team have any procedures in place for when such a critical bug is found? There are many people running this "stable" 4.19 branch, many of whom are unaware of this bug. Shouldn't the stable branch be rolled back to the last known good version? Going back to 4.18 is certainly a better option, but people unaware of this bug might still be running 4.19.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

James Courtier-Dutton ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #118 from James Courtier-Dutton ([email protected]) --- I have not observed the problem, but I have been thinking of maybe a more reliable way to detect a problem.
btrfs has a "scrub" command that essentially verifies the checksum of every file on the disk.
Now, ext4 does not have such a feature (as far as I know).
How about people who are seeing this problem, do a recursive sha1sum -b of every file on the disk while in a known good state, and then do a sha1sum -c of every file on the disk to see which ones got corrupted.
This might help when doing git bisect and checking that we are back to a known good file system, and in cases like comment #116, item 2.

Also, I think there is a way to force a reboot to a particular kernel, using grub, so one could script and git bisect, reboot to old working kernel, fsck, then reboot to problem kernel and start next git bisect all using automated scripts.

Anyway, just ideas.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #136 from [email protected] --- (In reply to Rainer Fiebig from comment #132)

I'm configuring the kernels myself. Two things that I always enable and that _might_ be related are:

CONFIG_BLK_WBT / CONFIG_BLK_WBT_MQ
and
CONFIG_CFQ_GROUP_IOSCHED

Maybe I can come up with a way to reproduce this bug more quickly. Writing a lot of (small) files and then deleting them seems like a good way so far.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #119 from Guenter Roeck ([email protected]) ---
I think I need some education. It has been suggested suggested several times - both here and elsewhere on the web - that the problem might possibly be caused by bad drives. Yet, I don't recall a single instance where a disk error was reported in conjunction with this problem. I most definitely don't see one on my systems.

Can hard drives and SSDs nowadays fail silently by reading bad data instead of reporting read (and/or write) errors ? I would find that thought quite scary. Can someone point me to related literature ?

In this context, it seems odd that this presumed silent disk error would only show up with v4.19.x, but not with earlier kernels.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #137 from Marc Burkhardt ([email protected]) --- (In reply to jaapbuurman from comment #135)

That would mean depublishing of the 4.19 release as a whole as nobody knows _what_ exactly to roll back. And if one would know, they would fix the bug instead.

I cannot remember such a scenario/bug in the past...

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #138 from Reindl Harald ([email protected]) ---
at least continue security updates for 4.18.x would probably be a good idea

Fedora 28 is already on 4.19.x

i run now 4.18.20-100.fc27.x86_64 which was the last Fedora 27 update on every F28 server and until this problem is solved i refuse to run 4.19.x in production which essentially means no security fixes for a unknown amount of time

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #120 from Florian Bruhin ([email protected]) ---

FWIW, https://github.com/claudehohl/checksummer does that (and saves the checksums in a sqlite database).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #121 from [email protected] ---
I have the problem again with kernel 4.19.5 (J. Axboe patches). Sorry I don't trust 4.19.x without Axboe patch because if some fs corruptions reappear they will be less violent than without the patch.

OS uptime 8:51, 37 static ext4 mountpoints as they are reported by 'mount' command.

I have checked with the same kernel 4.19.5 in rescue mode (i.e none of the file systems are mounted)

summary:
- e2fsck -fy /dev/dm-4 > log.1
- echo 3 > /proc/sys/vm/drop-caches
- e2fsck -fn /dev/dm-4 log.2
- reboot again in normal mode ( / is mounted)
- fsck -TRAC -r -M -p /dev/dm-X || fsck -f -TRAC -y -s /dev/dm-X
- if there is a new stable release then I compile and test the new release

Result:
The system doesn't see any error (fsck) on reboot (rescue and normal boot).

see attachments.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #122 from [email protected] ---
Created attachment 279771
--> https://bugzilla.kernel.org/attachment.cgi?id=279771&action=edit
dmesg shows errors before reboot

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #123 from [email protected] ---
Created attachment 279773
--> https://bugzilla.kernel.org/attachment.cgi?id=279773&action=edit
logs show no error after reboot

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #139 from Marc Burkhardt ([email protected]) --- (In reply to carlphilippreh from comment #136)

I have these config options set and _currently_ no corruption.

Having this compiled in is *probably* not what to look for. Rather people should seek for actual *usage* of these features. I use the deadline scheduler.

root@marc:~ # egrep "(BLK_WBT|IOSCH)" /boot/config-4.19.5loc64 CONFIG_BLK_WBT=y
CONFIG_BLK_WBT_SQ=y
CONFIG_BLK_WBT_MQ=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y CONFIG_DEFAULT_IOSCHED="deadline" CONFIG_MQ_IOSCHED_DEADLINE=y CONFIG_MQ_IOSCHED_KYBER=y
CONFIG_IOSCHED_BFQ=y
# CONFIG_BFQ_GROUP_IOSCHED is not set

Could someone gather a list of what actually is in .configs but is relevant/irrelevant? I don't want to do is but I'm not really sure to not mess is up. I mean there was "DAX enabled in the .config" talked about but I have it compiled-in but I'm not actually using is.

I would, moreover, like to gather actual setting used by people who run into the bug and those who are not, like currently used schedulers, nr_requests, discard, ...

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #124 from [email protected] ---
Please read more violent without the patch

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #140 from [email protected] ---
(In reply to Marc Burkhardt from comment #137)

I know it sounds bad, but isn't depublishing 4.19 the best course of action right now? There's probably a lot of people running 4.19 that are completely unaware of this bug and might or might not run into this later.

Data corruption issues are one of the worst, and should be addressed ASAP, even if it means temporary depublishing the latest kernel, right?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #141 from Marc Burkhardt ([email protected]) --- (In reply to jaapbuurman from comment #140)

4.18.20 is from Nov 21st and came with 4.19.3. It lacks 3 releases of fixes parallel to 4.19.6 due to 4.18 being EOL.

4.19 is out in the wild now. You cannot "get it back" ...

And people are probably more aware of a new 4.19 release pushed by the distros than a rollback of the 4.19 release.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #125 from Bart Van Assche ([email protected]) --- (In reply to James Courtier-Dutton from comment #118)

That could take a lot of CPU time. On my system git status told me that about ten source files had disappeared from the kernel tree that I definitely had not deleted myself. In other words, git can be used to detect filesystem corruption.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #142 from Rainer Fiebig ([email protected]) --- (In reply to carlphilippreh from comment #134)

Thanks. It'll take a while to sift through this.

As an alternative to 4.19 you may want to use one of the latest LTS-kernels, 4.14.84 perhaps.[1]
But before compiling/installing it, make sure the fs is OK (s. comment 114).

[1] https://www.kernel.org/

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Daniel Harding ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #143 from Daniel Harding ([email protected]) --- Another datapoint: I have observed Ext4 metadata corruption under both 4.19.1 and 4.19.4. I'm using LVM (but no RAID); the underlying drive is a 1GB SATA-attached Samsung 850 PRO SSD. I've not been able to reliably reproduce, but an rsync-based backup of my home partition runs once an hour and it usually starts reporting corruption errors within a day or two of booting a 4.19.x kernel. So far the corruption has only happened in directories that I am not actively using - as far as I know they are only being accessed by the rsync process. Since I started seeing the corruption under 4.19.x, I've run 4.18.16 for two stretches, one of which was twelve days, without any problems, so I'm quite confident it is not an issue of defective hardware. I have a weekly cron job which runs fstrim, but at least once I booted into 4.19.4 (previous boot was 4.18.16), and started seeing metadata corruption after about 36 hours, but fstrim had not run during that time.

Some (possibly) relevant kernel configs:
CONFIG_SCSI_MQ_DEFAULT=y
# CONFIG_DM_MQ_DEFAULT is not set
# CONFIG_MQ_IOSCHED_DEADLINE is not set
# CONFIG_MQ_IOSCHED_KYBER is not set
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
# CONFIG_DEV_DAX is not set
# CONFIG_FS_DAX is not set

$ cat /sys/block/sda/queue/scheduler
[none] bfq

I'm happy to report any more info about my kernel/system if it would be helpful, but unfortunately I don't have the bandwidth to do any bisection right now.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #144 from Rainer Fiebig ([email protected]) --- (In reply to Daniel Harding from comment #143)

Bisecting just fs/ext4 (comment 79) wouldn't cost much time. Just 32 commits, 5 steps. It won't get much cheaper than that.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #145 from Rainer Fiebig ([email protected]) --- (In reply to Reindl Harald from comment #138)

Perhaps you can use one of the LTS-kernels, like 4.14.84.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #146 from Reindl Harald ([email protected]) ---

on Fedora 28?
seriously?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #147 from Rainer Fiebig ([email protected]) --- (In reply to Reindl Harald from comment #131)

I've installed 4.19.x with a defconfig in a VirtualBox-VM, hoping the issue would show up and I could bisect it there. I've also varied the config-params that have been discussed here.

But unfortunately that damn thing runs as nicely in the VM as it does on real iron - at least here. :)

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #148 from Rainer Fiebig ([email protected]) --- (In reply to Reindl Harald from comment #146)

Sorry, just trying to help. And I didn't know that one can't run LTS-kernels on Fedora 28.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #149 from Jens Axboe ([email protected]) --- (In reply to Jimmy.Jazz from comment #121)

You had the issue with the full block patch applied, the one that includes both the synchronize_rcu() and the quiesce? Or just the partial one I suggested earlier?

Interesting, so it didn't make it to media.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Jukka Santala ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #150 from Jukka Santala ([email protected]) ---
Whole block corrupted at once with each inode/file returning "Structure needs cleaning" and bad extra_isizes in syslog while three hours in doing plain cp -ax from ext4 to BTRFS mdraid in Ubuntu 18.04.1 mainline kernel 4.19.6 in init level 1 (Ubuntu rescue mode with almost nothing else running).

Saved the corrupted block via debugfs, filesystem mounts read-only, dropped disk caches and block is fine again with files accessible. Seeing if I can find the corrupted block contents anywhere else in the filesystem. Error first happened for me in 4.19.5 after running cleanly for days, now it comes constantly.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #151 from [email protected] ---
(In reply to Jens Axboe from comment #149)

synchronize_rcu() and the quiesce as you asked me.

The following tests has be made on an other computer named orca to not be confused with earlier comments I have posted.

Again, I can confirm it but only with your patches applied. On orca with 4.20 and w/o your patch the bug was able to entirely wipe out orca postgres database :(

It gives me the opportunity to do a full reinstall of orca from the stick. Don't get confused with mmp_node_name host name on the new created partitions, it has an easy explanation. The bootable stick used to create the filesystems has a different hostname than the final server (i.e. orca)

Please read the attached bug.orca.tar.xz tar file. You can follow the logs sequence from the file creation time.

I underline, the new corruption on dm-10 after the server has rebooted has nothing to do with the one announced earlier in dmesg. Read dmesg-zalman.txt, dmesg-zalman-2.txt and dumpe2fs-dm-10-after-e2fsk.txt, dmesg-after-e2fsk.txt in that order. It shows that dm-10 corruption was initiate during the reboot.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #152 from [email protected] ---
Created attachment 279801
--> https://bugzilla.kernel.org/attachment.cgi?id=279801&action=edit
new generated server

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #153 from James Courtier-Dutton ([email protected]) --- (In reply to Theodore Tso from comment #127)

Is it possible to determine the locality of these corruptions? I.e. Is the corruption to a contiguous page of data (e.g. 4096 bytes corrupted) or is the corruption scattered, a few bytes here, a few bytes there?

YYY" can I assume this is fact, or is it still TBD?

If it is contiguous data, is there any pattern to the data that would help us identify where it came from?

Maybe that would help work out where the corruption was coming from. Maybe it is DMA from some totally unrelated device driver, but by looking at the data, we might determine which device driver it is?
It might be some vulnerability in the kernel that some hacker is trying to exploit, but unsuccessfully, resulting in corruption. This could explain the reason why more people are not seeing the problem.

Some people reporting that the corruptions are not getting persisted to disk in all cases, might imply that the corruption is happening outside the normal code paths, because the normal code path would have tagged the change as needing flushing to disk at some point.

Looking at the corrupted data would also tell us if values are within expected ranges, that the normal code path would have validated. If they are outside those ranges, then it would again imply that the corrupt data is not being written by the normal ext4 code path, thus further implying that there is not a bug in the ext4 code, but something else in the kernel is writing to it by mistake.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #154 from James Courtier-Dutton ([email protected]) --- I have scanned all the comments. So far I have only seen 1 person who has this problem and have also reported what hardware they have.
So, the sample size is statistically far too small to conclude that it is an AMD or a INTEL only problem.
Is there anyone out there who sees this problem, and is running Intel hardware? How many people are seeing this problem? Can they each post the output of "lspci -vvv" and a dmesg showing the problem they have?

This appears to be a problem that is reported by an extremely small amount of people.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #155 from Rainer Fiebig ([email protected]) --- #154 re. Intel: start with comments 3/5/6.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #156 from Guenter Roeck ([email protected]) --- Status update: I have not been able to reproduce the problem with v4.19.6 minus the reverts from #99. I did see some failures, specifically exactly one per affected file system, but I attribute those to false positives (I did not run fsck as recommended prior to starting the test).

stats:

System 1:

uptime: 18h 45m

iostats:
Device tps MB_read/s MB_wrtn/s MB_read MB_wrtn

loop0             0.00         0.00         0.00          0          0
nvme0n1         195.54         3.27         3.96     220723     267555
sda             128.88         0.36        18.91      24659    1277907

sdb 131.40 18.85 5.07 1273780 342404

nvme0n1 and sda were previously affected. sdb is a new drive.

System 2:

uptime: 14h 56m

iostats:
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
loop0 0.00 0.00 0.00 5 0
sda 26.45 538.25 87.87 28965283 4728576
sdb 108.87 2875.25 4351.42 154728917 234167724

Both sda and sdb were previously affected.

My next step will be to try v4.19.6 with the following reverts:

Revert "ext4: handle layout changes to pinned DAX mappings" Revert "dax: remove VM_MIXEDMAP for fsdax and device dax" Revert "ext4: close race between direct IO and ext4_break_layouts()"

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Michael Duell ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #157 from Michael Duell ([email protected]) ---
I have had file system corruption with 4.19 on a BTRFS file system as well. 4.19.2 Kernel. I think this has to be related. No files were actually corrupted but the Kernel set the file system read-only as soon as the error occurred.

I have even tried a NEW and FRESH btrfs file system created via a LiveCD system and it happened there as well as soon as I did a btrfs send/receive operation.

I am on a Thinkpad T450.

lspci -vvv
https://paste.pound-python.org/show/9tZLWlry0Iy7Z629VPea/

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #158 from Michael Duell ([email protected]) --- Same paste as root:

https://paste.pound-python.org/show/DZTBYXQBFhcHi69OBba8/

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #159 from Rainer Fiebig ([email protected]) --- #156: A ray of hope. Underpins Nestors findings (comment 78).

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #160 from Guenter Roeck ([email protected]) ---

Another update: I hit the problem almost immediately with the reverts from #156.

[ 1826.738686] EXT4-fs error (device sda1): ext4_iget:4796: inode #7633436:
comm borg: bad extra_isize 28534 (inode size 256)
[ 1826.740744] Aborting journal on device sda1-8.
[ 1826.747339] EXT4-fs (sda1): Remounting filesystem read-only

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #161 from Rainer Fiebig ([email protected]) --- #160: Down to five now.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #162 from Guenter Roeck ([email protected]) --- As next step, I am going to try v4.19.6 with the following reverts:

Revert "ext4: readpages() should submit IO as read-ahead" Revert "ext4: improve code readability in ext4_iget()"

Those with btrfs problems might consider reverting commit 5e9d398240b2 ("btrfs:
readpages() should submit IO as read-ahead") and report the results.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #163 from Jukka Santala ([email protected]) ---
First report specifies AMD, second report is Intel, and so on. I agree more detailed system information might help find commonalities and false positives, but the cross-platform nature of the problem seemed established right from the start.

AMD Phenom(tm) II X4 B50 Processor, SSHD ST1000LM014-1EJ164

[11514.358542] EXT4-fs error (device dm-0): ext4_iget:4831: inode #18288150:
comm cp: bad extra_isize 49917 (inode size 256)
[11514.386613] Aborting journal on device dm-0-8.
[11514.389070] EXT4-fs (dm-0): Remounting filesystem read-only

Errors for each of the inodes on the block follow, until I dropped filesystem caches (drop_caches 3) and accessed them again and they were fine. Corrupted block looked random binary, but not compressed. BTRFS was reporting csum errors every time I dropped caches, which makes me wonder if people having the problem are using BTRFS?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Andreas Dilger ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |adilger.kernelbugzilla@dilg
                   |                            |er.ca

--- Comment #164 from Andreas Dilger ([email protected]) --- There was a recent post on linux-ext4 that this might relate to a compiler bug:

Can people hitting this please confirm or deny whether this compiler is in use on your system.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #165 from Guenter Roeck ([email protected]) ---

groeck@server:~$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #166 from Jukka Santala ([email protected]) --- However, at least here:
cat /proc/version
Linux version 4.19.6-041906-generic (kernel@gloin) (gcc version 8.2.0 (Ubuntu 8.2.0-9ubuntu1)) #201812010432 SMP Sat Dec 1 09:34:07 UTC 2018

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #167 from Bart Van Assche ([email protected]) --- All 4.19.x kernels I tested were built with gcc version 8.2.1 20181025 [gcc-8-branch revision 265488].

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Michel Roelofs ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #168 from Michel Roelofs ([email protected]) --- Hereby my experience that may be related:

[ 2451.982816] EXT4-fs error (device dm-1): ext4_iget:4831: inode #6029313:
comm ls: bad extra_isize 5 (inode size 256)
root@ster:/# debugfs -R 'ncheck 6029313' /dev/dm-1
debugfs 1.43.4 (31-Jan-2017)
Inode   Pathname
6029313 //sabnzb
ncheck: Inode checksum does not match inode while doing inode scan

root@ster:/# echo 2 > /proc/sys/vm/drop_caches
root@ster:/# debugfs -R 'ncheck 6029313' /dev/dm-1
debugfs 1.43.4 (31-Jan-2017)
Inode   Pathname
6029313 //sabnzb
ncheck: Inode checksum does not match inode while doing inode scan

root@ster:/#  echo 3 > /proc/sys/vm/drop_caches
root@ster:/# debugfs -R 'ncheck 6029313' /dev/dm-1
debugfs 1.43.4 (31-Jan-2017)
Inode   Pathname
6029313 //sabnzb

Kernel v4.19.5, CPU Intel Atom D525, Debian Linux 9.6, brand new WDC WD40EFRX-68N32N0, gcc 6.3.0-18+deb9u1.

Also seen with an ext4 filesystem created on Nov 21 2018.

Also seen with earlier 4.19.0 kernel, and older WDC WD30EFRX-68A in same computer.

Going back to v4.18.<latest> kernel solved the issues. No disk corruption shown by e2fsck.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Sune Mølgaard ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #169 from Sune Mølgaard ([email protected]) --- FWIW, I didn't see any problems with 4.19.0, but see it on all my systems with 4.19.3 and onward (although I *did* skip 4.19.[12].

There fore, I embarked on a git bisect in linux-stable from v4.19 to v4.19.3, which is nearing its end, *so far with every iteration marked GOOD*.

Referencing https://www.spinics.net/lists/linux-ext4/msg63498.html (#164), and noting that I usually run kernels from kernel.ubuntu.com/~kernel-ppa/mainline , I did the following:

smo@dell-smo:~$ cat /proc/version 
Linux version 4.19.0-041900-generic (kernel@tangerine) (gcc version 8.2.0
(Ubuntu 8.2.0-7ubuntu1)) #201810221809 SMP Mon Oct 22 22:11:45 UTC 2018

Then, I downloaded 4.19.3 from kernel-ppa, unpacked, and:

smo@dell-smo:~/src/deb/foo/boot$ strings vmlinuz-4.19.3-041903-generic |grep
8.2.0
4.19.3-041903-generic (kernel@gloin) (gcc version 8.2.0 (Ubuntu
8.2.0-9ubuntu1)) #201811210435 SMP Wed Nov 21 09:37:20 UTC 2018

BANG, as they say: 8.2.0-9.

Whereas git bisect "GOOD"s continuously (as stated, it is not complete - only almost) are not impossible, they certainly don't seem entirely normal, but:

sune@jekaterina:~$ gcc --version
gcc (Ubuntu 8.2.0-7ubuntu1) 8.2.0

...on the system where I self-compile during the bisect, *could* explain it.

My impression is, that a lot of affected people are on Ubuntu, and I suspect the following:

* Many of the affected Ubuntu folks do indeed use kernels from kernel-ppa

* Some of those, as well as non-Ubuntu-folks, may have that compiler version for other reasons, and hit the bugs on that account

* Bisecting yields inconclusive results, as it seems to do for me, since the Issues is non-kernel.

* Theodore T'so and Jens Axboe are unable to reproduce due to unaffected compiler versions, which also explains the no-show in regression tests.

Tso, Axboe: With the two of you being completely unable to replicate, could you be enticed to either try GCC 8.2.0-9 (or, possibly, just the packages from the following URLs, and run your regression tests against those?

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19 (presumed GOOD)

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19 (presumed BAD)

Best regards,

Sune Mølgaard

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #170 from Sune Mølgaard ([email protected]) --- Meh, typos:

1. "(or, possibly..." should end the parenthesis after "...the following URLs)

2. Last link should be http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.3

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #171 from Guenter Roeck ([email protected]) ---
To reiterate, I use gcc version "5.4.0-6ubuntu1~16.04.10" to build my kernels. Also, I build the kernels on a system not affected by the problem. It may well be that a compiler problem in gcc 8.2.0 causes additional trouble, but it is not the trouble observed on my affected systems.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #172 from Michael Orlitzky ([email protected]) ---

$ gcc --version
gcc (Gentoo Hardened 7.3.0-r3 p1.4) 7.3.0

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Lei Ming ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #173 from Lei Ming ([email protected]) ---
The commit 2a5cf35cd6c56b2924("block: fix single range discard merge") in linus tree may address one possible data loss, anyone who saw corruption in scsi may try this fix and see if it makes a difference.

Given the merged discard request isn't removed from elevator queue, it might be possible to be submitted to hardware again.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #174 from Theodore Tso ([email protected]) ---
One of the reasons why this is bug hunt is so confounding. While I was
looking at older reports to try to see if I could find common factors, I found Jimmy's dmesg report in #50, and this one looks different from many of the
others that people have reported. In this one, the EXT4 errors are preceeded
by a USB disconnect followed by disk-level errors.

This is why it's important that we try very hard to filter out false positives and false negative reports. We have multiple reports which both strongly indicate that it's an ext4 bug, and others which strongly indicate it is a bug
below the file system layer. And then we have ones like this which look like
a USB disconnect....

[52967.931390] usb 4-1: reset SuperSpeed Gen 1 USB device number 2 using
xhci_hcd
[52968.985620] sd 8:0:0:2: [sdf] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00
driverbyte=0x08
[52968.985624] sd 8:0:0:2: [sdf] tag#0 Sense Key : 0x6 [current] 
[52968.985626] sd 8:0:0:2: [sdf] tag#0 ASC=0x28 ASCQ=0x0 
[52968.985628] sd 8:0:0:2: [sdf] tag#0 CDB: opcode=0x2a 2a 00 00 cc 60 28 00 08
00 00
[52968.985630] print_req_error: I/O error, dev sdf, sector 13393960
[52968.985641] EXT4-fs warning (device sdf2): ext4_end_bio:323: I/O error 10
writing to inode 522 (offset 6132072448 size 6295552 starting block 1674501)
[52968.985643] Buffer I/O error on device sdf2, logical block 1673728
[52968.985651] Buffer I/O error on device sdf2, logical block 1673729
[52968.985654] Buffer I/O error on device sdf2, logical block 1673730
[52968.985659] Buffer I/O error on device sdf2, logical block 1673731
[52968.985663] Buffer I/O error on device sdf2, logical block 1673732
...
[52968.986231] EXT4-fs warning (device sdf2): ext4_end_bio:323: I/O error 10
writing to inode 522 (offset 6132072448 size 8388608 starting block 1675013)
[52969.435367] JBD2: Detected IO errors while flushing file data on sdf2-8
[52969.435407] Aborting journal on device sdf2-8.
[52969.435422] JBD2: Error -5 detected when updating journal superblock for
sdf2-8.
[52969.441997] EXT4-fs error (device sdf2): ext4_journal_check_start:61:
Detected aborted journal
[52985.065239] EXT4-fs error (device sdf2): ext4_remount:5188: Abort forced by

user

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #175 from Sune Mølgaard ([email protected]) ---
I guess I may have been biased towards the posts mentioning the GCC bug, then, but that would lead me to think that I am not alone in conflating that one with actual ext4 or block layer bugs.

I shall go ahead and reference my comment above (#169) to the Ubuntu kernel-ppa folks, and in the event that this will then preclude others from mis-attributing the GCC bug to these, I should hope to at least effect an elimination of that noise source from this Bugzilla entry.

My apologies, and keep up the good work!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #176 from Theodore Tso ([email protected]) ---
Hi Sune, alas for your theory in #169, I am already using gcc 8.2.0-9 from Debian testing.

% gcc --version
gcc (Debian 8.2.0-9) 8.2.0

Could it an Ubuntu-specific issue? I don't think so, since there have been
some people running Debian and Gentoo who have reported the problem, and one person who reported the problem was running Debian and was using gcc 8.2.0-9.

I have built kernels using gcc 8.2.0-9 and used them for regression testing using gce-xfstests:

% objdump -s --section .comment /build/ext4-64/vmlinux

Contents of section .comment:
0000 4743433a 20284465 6269616e 20382e32 GCC: (Debian 8.2
0010 2e302d39 2920382e 322e3000 .0-9) 8.2.0.

The kernel I am using on my personal development laptop was compiled using gcc 8.2.0-8:

% objdump -s --section .comment /usr/lib/debug/lib/modules/4.19.0-00022-g831156939ae8/vmlinux

Contents of section .comment:
0000 4743433a 20284465 6269616e 20382e32 GCC: (Debian 8.2
0010 2e302d38 2920382e 322e3000 .0-8) 8.2.0.

Of course, I'm not doing anything more exciting than running chrome, mutt, emacs, and building kernels most of the time...

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #177 from Néstor A. Marchesini ([email protected]) --- I did a lot of tests here, the first thing was to configure with tune2fs so that in each boot I forcefully check my three partitions, the / boot the / root and the / home partition.

# tune2fs -c 1 /dev/md0
# tune2fs -c 1 /dev/md2
# tune2fs -c 1 /dev/md3

I have reinstalled and compiled tree 4.19.5 and 4.19.0 from scratch, as well as the new tree 4.19.6.
I have not had problems with the 4.19.5 or with the new 4.19.6, many hours of use and restarts every time .. everything perfect.
But at the first boot with 4.19.0 ... corruption of the root partition. it leaves me in the console for repair, I repair it with:

# fsck.ext4 -y /dev/md2

After started, I'll see /lost+found and find many folders and files in perfect condition, not corrupt, but with the numeric names #

# ls -l /lost+found/
total 76
-rw-r--r-- 1 portage portage 5051 dic 10  2013 '#1057825'
drwxr-xr-x 3 portage portage 4096 dic 10  2013 '#1057827'
-rw-r--r-- 1 root    root    2022 oct 22 03:37 '#3184673'
-rw-r--r-- 1 root    root     634 oct 22 03:37 '#3184674'
etc...
etc...

So decided I started with the bisection, download only from 4.18 onwards.

$ su
# cd /usr/src
# git clone
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
--shallow-exclude v4.17 linux-stable

# eselect kernel list
Available kernel symlink targets:
  [1]   linux-4.18.20-gentoo
  [2]   linux-4.19.0-gentoo
  [3]   linux-4.19.5-gentoo
  [4]   linux-4.19.6-gentoo *
  [5]   linux-stable

# eselect kernel set 5

# eselect kernel list 
Available kernel symlink targets:
  [1]   linux-4.18.20-gentoo
  [2]   linux-4.19.0-gentoo
  [3]   linux-4.19.5-gentoo
  [4]   linux-4.19.6-gentoo
  [5]   linux-stable *

# ls -l
total 20
lrwxrwxrwx  1 root root   12 dic  2 21:27 linux -> linux-stable
drwxr-xr-x 27 root root 4096 nov 24 14:44 linux-4.18.20-gentoo
drwxr-xr-x 27 root root 4096 dic  2 20:28 linux-4.19.0-gentoo
drwxr-xr-x 27 root root 4096 dic  2 03:47 linux-4.19.5-gentoo
drwxr-xr-x 27 root root 4096 dic  2 14:50 linux-4.19.6-gentoo
drwxr-xr-x 26 root root 4096 dic  2 19:18 linux-stable

# cd linux
# git bisect start v4.19 v4.18 -- fs/ext4
Bisectando: faltan 16 revisiones por probar después de esto (aproximadamente 4
pasos)
[863c37fcb14f8b66ea831b45fb35a53ac4a8d69e] ext4: remove unneeded variable "err"
in ext4_mb_release_inode_pa()

# git bisect log
# bad: [84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d] Linux 4.19
# good: [94710cac0ef4ee177a63b5227664b38c95bbf703] Linux 4.18
git bisect start 'v4.19' 'v4.18' '--' 'fs/ext4'

Just beginning, today was Sunday and ... besides little experience with git :) I was also looking at the ebuilds of the gentoo-sources trees to know what patches I applied to emerge when installing the sources.

$ cat /usr/portage/sys-kernel/gentoo-sources/gentoo-sources-4.18.20.ebuild
|grep K_GENPATCHES_VER=
K_GENPATCHES_VER="24"
$ ls -lh /usr/portage/distfiles/genpatches-4.18-24.base.tar.xz 
-rw-rw-r-- 1 portage portage 661K nov 21 10:13
/usr/portage/distfiles/genpatches-4.18-24.base.tar.xz
$ tar -tf /usr/portage/distfiles/genpatches-4.18-24.base.tar.xz 
./0000_README
./1000_linux-4.18.1.patch
./1001_linux-4.18.2.patch
./1002_linux-4.18.3.patch
./1003_linux-4.18.4.patch
./1004_linux-4.18.5.patch
./1005_linux-4.18.6.patch
./1006_linux-4.18.7.patch
./1007_linux-4.18.8.patch
./1008_linux-4.18.9.patch
./1009_linux-4.18.10.patch
./1010_linux-4.18.11.patch
./1011_linux-4.18.12.patch
./1012_linux-4.18.13.patch
./1013_linux-4.18.14.patch
./1014_linux-4.18.15.patch
./1015_linux-4.18.16.patch
./1016_linux-4.18.17.patch
./1017_linux-4.18.18.patch
./1018_linux-4.18.19.patch
./1019_linux-4.18.20.patch
./1500_XATTR_USER_PREFIX.patch
./1510_fs-enable-link-security-restrictions-by-default.patch
./2500_usb-storage-Disable-UAS-on-JMicron-SATA-enclosure.patch
./2600_enable-key-swapping-for-apple-mac.patch

$ tar -xf /usr/portage/distfiles/genpatches-4.18-24.base.tar.xz
./1019_linux-4.18.20.patch
$ ls -lh 1019_linux-4.18.20.patch
-rw-r--r-- 1 nestor nestor 164K nov 21 10:01 1019_linux-4.18.20.patch

$ cat /usr/portage/sys-kernel/gentoo-sources/gentoo-sources-4.19.0.ebuild |grep
K_GENPATCHES_VER=
K_GENPATCHES_VER="1"
$ ls -lh /usr/portage/distfiles/genpatches-4.19-1.base.tar.xz
-rw-rw-r-- 1 portage portage 4,0K oct 22 08:47
/usr/portage/distfiles/genpatches-4.19-1.base.tar.xz
$ tar -tf /usr/portage/distfiles/genpatches-4.19-1.base.tar.xz
./0000_README
./1500_XATTR_USER_PREFIX.patch
./1510_fs-enable-link-security-restrictions-by-default.patch
./2500_usb-storage-Disable-UAS-on-JMicron-SATA-enclosure.patch
./2600_enable-key-swapping-for-apple-mac.patch

As you can see in the 4.19.0 tree do not apply patches 1000_linux-4.19.x.patch

My gcc version for quite some time:

$ gcc -v
gcc versión 8.2.0 (Gentoo 8.2.0-r5 p1.6

Obviously something happens with the inodes, but apparently only I'm doing it now with the tree 4.19.0.
If I find something I will be reporting it.

Regards

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #178 from Sune Mølgaard ([email protected]) --- Hi Theodore,

I am not much of a kernel developer, let alone and FS one, so your guesses would be vastly better founded than mine.

I could imagine, though, that a combination of GCC version, .config and, possibly, the creation time (kernel version-wise) of the FSs in question, could create a sort of "cocktail effect". For my part, none of my FSs are < at least a year old.

FWIW, I started seeing the problem specifically with 4.19.3 (4.19.0 being good, and built with 8.2.0-7), but that was after skipping 4.19.[12].

I note that the first Ubuntu kernel-ppa kernel the be built with 8.2.0-9 was 4.19.1, so if my ongoing bisect ends without any triggering of the bug I see, I shall try kernel-ppa 4.19.1 - if that exhibits the bug, then that further points to GCC, but as you say, perhaps specifically for the Ubuntu kernels.

Now, as someone else stated somewhere, the only things that kernel-ppa patches, are some Ubuntu-specific build and package structure, as well as .config hte lst part being available at
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.1/0006-configs-based-on-Ubuntu-4.19.0-4.5.patch
.

As promised above, I have written the kernel-ppa team lead, Bradd Figg, and I would expect him and his team to be better at pinpointing which combination of GCC 8.2.0-9 and .config options might be problematic, but if they find that the problem goes away with a GCC upgrade, they might opt for letting that be it.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #179 from Theodore Tso ([email protected]) ---
Michael Orlitzky: In your report, you've indicated that you've only been seeing bugs in files that are being *read* and that these were files that were written
long ago. If you reboot, or drop caches using "echo 3 >
/proc/sys/vm/drop_caches" do the files stay corrupted? Some of the reports
(but not others) seem to indicate the problem is happening on read, not on write. Of course, some of the reports are relating to metadata blocks getting corrupted on read, while your report is about data blocks getting reported on read.

Thanks!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #180 from Eric Benoit ([email protected]) ---
I've been able to reproduce the issue on the other workstation I'd mentioned earlier with ZFS:

NAME                        STATE     READ WRITE CKSUM
nipigon                     ONLINE       0     0     0 
  mirror-0                  ONLINE       0     0     0 
    wwn-0x5000c5006407e87e  ONLINE       0     0    17 
    wwn-0x5000c5004e4e92a9  ONLINE       0     0     6 

This isn't particularly hard to trigger; just a bunch of files filled with /dev/urandom (64M*14000) being read back concurrently (eight processes) over about 90 minutes.

Kernel version is 4.19.6 compiled with gcc 8.2.0 (Gentoo 8.2.0-r5 p1.6).

Have we any other ZFS users experiencing this?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #181 from Guenter Roeck ([email protected]) --- #180: Eric, would you mind sharing the script used to create the files and to read them back ?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #182 from Guenter Roeck ([email protected]) ---
As for where problems are seen, for my part the problem is seen mostly when trying to read files created recently as part of kernel builds. The problems are reported with reads, but writes are definitely involved, at least for me.

As for blaming gcc, or Ubuntu, or both, I would kindly like to remind people that I see the problem on two systems out of four running v4.19.x kernels, all with the same kernel build and configuration.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #183 from Theodore Tso ([email protected]) ---
Eric, re #180, could you upload your .config file for your kernel and the
boot-command line? I'm interested in particular what I/O scheduler you are
using. And are you using the same .config and boot command line (and other
system configurations) on your other system where you were seeing the problem? Many thanks!!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #184 from Lei Ming ([email protected]) ---
The commit 2a5cf35cd6c56b2924("block: fix single range discard merge") in linus tree may address one possible data loss, anyone who saw corruption in scsi may try this fix and see if it makes a difference.

Given the merged discard request isn't removed from elevator queue, it might be possible to be submitted to hardware again.(In reply to Theodore Tso from comment #174)

IMO it should be a usb device reset instead of disconnect, and reset is often triggered in SCSI EH.

Thanks,

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #185 from Eric Benoit ([email protected]) --- (In reply to Guenter Roeck from comment #181)

Just a pair of trivial one-liners:

for i in {00000..13999}; do echo dd bs=1M count=64 if=/dev/urandom of=urand.$i; done

for i in urand.*; do echo dd bs=1M if=$i of=/dev/null; done | parallel -j8

I'm using /dev/urandom since I have lz4 compression enabled. I imagine /dev/zero would be just as effective if you don't.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #186 from Eric Benoit ([email protected]) --- Created attachment 279807
--> https://bugzilla.kernel.org/attachment.cgi?id=279807&action=edit
tecciztecatl linux kernel 4.19.6 .config

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #187 from Eric Benoit ([email protected]) --- (In reply to Theodore Tso from comment #183)

[    0.000000] Command line: BOOT_IMAGE=/root@/boot/vmlinuz root=simcoe/root

triggers=zfs radeon.dpm=1

# cat /sys/block/sd[a-d]/queue/scheduler    
[none] 
[none] 
[none] 
[none] 

The config between 4.18.20 and 4.19.6 are about as identical as possible, the only differences being whatever was added in 4.19 and prompted by make oldconfig.

Between this machine and the other (a server) the only differences would be in specific hardware support and options suitable for that application. In terms of schedulers, block devices, and filesystem support, they're the same.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #188 from Eric Benoit ([email protected]) --- (In reply to Eric Benoit from comment #185)

Er whoops, might want to remove the echo or pipe it to parallel.

I'm repeating things under 4.18.20 just for comparison. It's been about an hour now without a single checksum error reported.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #189 from Michael Orlitzky ([email protected]) --- (In reply to Theodore Tso from comment #179)

Each time the corruption has been reported by the backup job that I run overnight. When I see the failed report in the morning, I reboot into SystemRescueCD (which is running 4.14.x) and then run fsck to fix things. The fsck does indeed find a bunch of corruption, and appears to fix it.

The first couple of times I verified the corruption by running something like "git gc" in the affected directory, and IIRC I got the same "structure needs cleaning" error back. Before that, I hadn't touched that repo in a while. But since then, I've just been rebooting immediately and running fsck -- each time finding something wrong and (I hope) correcting it.

It takes about a week for the corruption to show up, but if there's some test you need me to run I can boot back into 4.19.6 and roll the dice.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #190 from Andreas John ([email protected]) --- (In reply to Michael Orlitzky from comment #189)

Hm, interesting that it happens to my Ubuntu Bionic on bare metal Macbook Pro (SSD) within minutes - even if I don't habe much I/O load. I am doing the fsck mit 4.18.20. What could be the difference?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #191 from Marc Burkhardt ([email protected]) --- (In reply to Eric Benoit from comment #185)

Don't know if my comments are relevant as I got no reply as of now but here some info regarding this test:

I ran it without errors on my /home partition wich is a dm-crypt ext4 setup using the deadline mq-scheduler and the gcc 8 compiler branch. The partition is mounted

/dev/mapper/crypt-home on /home
type ext4 (rw,nosuid,noatime,nodiratime,quota,usrquota,grpquota,errors=remount-ro)

[    0.000000] Linux version 4.19.6loc64 (marc@marc) (gcc version 8.2.0 (Gentoo
Hardened 8.2.0-r4 p1.5)) #1 SMP PREEMPT Sat Dec 1 16:00:21 CET 2018
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.19.6loc64 root=/dev/sda7 ro

init=/sbin/openrc-init root=PARTUUID=6d19e60a-72a8-ee44-89f4-cc6f85a9436c real_root=/dev/sda7 ro resume=PARTUUID=fbc25a25-2d09-634d-9e8b-67308f2feddf real_resume=/dev/sda8 acpi_osi=Linux libata.dma=3 libata.noacpi=0 threadirqs rootfstype=ext4 acpi_sleep=s3_bios,s3_beep devtmpfs.mount=0 net.ifnames=0 vmalloc=512M noautogroup elevator=deadline libata.force=noncq nmi_watchdog=0 i915.modeset=0 cgroup_disable=memory scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y vgacon.scrollback_persistent=1 processor.ignore_ppc=1 intel_iommu=igfx_off crashkernel=128M apparmor=1 security=apparmor nouveau.noaccel=0 nouveau.nofbaccel=1 nouveau.modeset=1 nouveau.runpm=0 nouveau.debug=disp=trace,i2c=trace,bios=trace nouveau.config=NvPmShowAll=true

[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel

Might be worth getting this guy aboard - got now reply though.

https://www.phoronix.com/forums/forum/software/general-linux-open-source/1063976-some-users-have-been-hitting-ext4-file-system-corruption-on-linux-4-19?p=1064826#post1064826

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #192 from Rainer Fiebig ([email protected]) --- #136

It has been suggested that I/O-schedulers may play a role in this. So here's are my settings for 4.19.x for comparison. They deviate from yours in some points but I really don't know whether this has any relevance. You may want to give it a try anyway. As I've said, 4.19.x is a nice kernel here.

CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_CGROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y CONFIG_RT_GROUP_SCHED=y CONFIG_SCHED_AUTOGROUP=y CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_MC_PRIO=y CONFIG_SCHED_HRTICK=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set # CONFIG_CPU_FREQ_GOV_SCHEDUTIL is not set # IO Schedulers
CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y CONFIG_CFQ_GROUP_IOSCHED=y CONFIG_DEFAULT_IOSCHED="deadline" CONFIG_MQ_IOSCHED_DEADLINE=y CONFIG_MQ_IOSCHED_KYBER=y
# CONFIG_IOSCHED_BFQ is not set CONFIG_NET_SCHED=y
# Queueing/Scheduling CONFIG_USB_EHCI_TT_NEWSCHED=y CONFIG_SCHED_INFO=y
CONFIG_SCHED_TRACER=y

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #193 from Ortwin Glück ([email protected]) ---
I am desperately trying to reproduce this in a qemu/KVM virtual machine with the configs given by those users. But until now to no avail.

If anybody has seen this issue in a VM please share your .config, mount options of all filesystems, kernel command line, and possibly workload that you are running.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #194 from Marc Burkhardt ([email protected]) --- (In reply to Rainer Fiebig from comment #192)

Really, how come you say "these are your settings"?

The settings are, what is actually being used not what has ben compiled-in or I miss anything?

What's the coincidence between

CONFIG_DEFAULT_IOSCHED="deadline" + CONFIG_IOSCHED_DEADLINE=y

and

cat /sys/block/sda/queue/scheduler
mq-deadline [kyber] bfq none

Please see #139 - wee need a list of what is effectively used and not what is actually possible. Bare metal or not. Intel? AMD? hugepages or nor?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #195 from Rainer Fiebig ([email protected]) --- #193

Good luck, Mr. Glück! ;)
My VirtualBox-VM seems immune to this issue. Perhaps VMs have just the right "hardware".

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #196 from Rainer Fiebig ([email protected]) --- #194

Hair-splitting won't help in this matter.

And btw: if you're so smart - how come you haven't solved this already?

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #197 from Marc Burkhardt ([email protected]) --- (In reply to Marc Burkhardt from comment #194)

I use an allegedly wrong compiler, I use 4.19.y, I use ext4 with and without dm-crypt, I use a scheduler, .... and I am currently NOT affected by that bug even running the tests that people say triggers the bug.

Just to make it clear again: I'm not a kernel dev, ok, but I use Linux for along time and I'm willing to help out what setup is *not* affected. Maybe I do something totally wrong here but I'm willing to help out as a gibe-back to the community providing me the OS I use solely for 20+ years.

I think the discussion should (at this point) not gather around what your kernel is *capable* of, but just what actually is set-up to trigger the bug.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

Sebastian Jastrzebski ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #198 from Sebastian Jastrzebski ([email protected]) --- I can also confirm the fs corruption issue on Fedora 29 with 4.19.5 kernel. I run it on ThinkPad T480 with NVME Samsung drive.

* Workload

The workload involves doing a bunch of compile sessions and/or running a VM (under KVM hypervisor) with NFS server. It usually takes anywhere from few hours to a day for the corruption to occur.

* Symptoms

- /dev/nvm0n1* entries disappear from /dev/
- unable to start any program as i get I/O errors

* System Info

Linux skyline.origin 4.19.5-300.fc29.x86_64 #1 SMP Tue Nov 27 19:29:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

BOOT_IMAGE=/vmlinuz-4.19.5-300.fc29.x86_64 root=/dev/mapper/fedora_skyline-root ro rd.lvm.lv=fedora_skyline/root rd.luks.uuid=luks-b66e85a5-f7b1-4d87-8fab-a01687e35056 rd.lvm.lv=fedora_skyline/swap rhgb quiet LANG=en_US.UTF-8

[none] mq-deadline

NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
nvme0n1 259:0 0 238.5G 0 disk
├─nvme0n1p1 259:1 0 200M 0 part
/boot/efi

├─nvme0n1p2                                   259:2    0     1G  0 part  /boot
├─nvme0n1p3                                   259:3    0   160G  0 part  
│ └─luks-b66e85a5-f7b1-4d87-8fab-a01687e35056 253:0    0   160G  0 crypt 
│   ├─fedora_skyline-root                     253:1    0   156G  0 lvm   /
│   └─fedora_skyline-swap                     253:2    0     4G  0 lvm   [SWAP]
└─nvme0n1p4                                   259:4    0  77.3G  0 part  
  ├─skyline_vms-atomic_00                     253:3    0    20G  0 lvm   
  └─skyline_vms-win10_00                      253:4    0    40G  0 lvm   

This is dumpe2fs output on the currently booted system.

dumpe2fs 1.44.3 (10-July-2018)

Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          410261f3-0779-455b-9642-d52800292fd7
Filesystem magic number:  0xEF53

Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype
needs_recovery extent 64bit flex_bg sparse_super large_file h uge_file uninit_bg dir_nlink extra_isize

Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              10223616
Block count:              40894464
Reserved block count:     2044723
Free blocks:              26175785
Free inodes:              9255977
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Mon Feb 19 18:48:05 2018
Last mount time:          Mon Dec  3 08:07:30 2018
Last write time:          Mon Dec  3 03:07:29 2018
Mount count:              137
Maximum mount count:      -1
Last checked:             Sat Jul 14 07:11:08 2018
Check interval:           0 (<none>)
Lifetime writes:          1889 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       9318809
Default directory hash:   half_md4
Directory Hash Seed:      ad5a6f9c-6250-4dc5-84d9-4a3b14edc7b7
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke journal_64bit
Journal size:             1024M
Journal length:           262144
Journal sequence:         0x00508e50
Journal start:            1

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685

--- Comment #199 from Ortwin Glück ([email protected]) --- (In reply to Rainer Fiebig from comment #195)

Yeah, I've been wondering. Qemu only exposes single-queue devices (virtio_blk) so bugs in MQ can not trigger here I guess. Also "hardware" timing is much different in VMs so race conditions may not trigger with the same frequency. Also CPU assignment/scheduling may be different with respect to barriers, so memory safety problems (RCU bugs, missing barriers) may behave differently.

I had no luck with vastly overcommitting vCPUs either.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=201685