Skip to contentSkip to conversation

ext4 corruption on parisc since 6.12

bisected matoro 4 participants 9 messages Dec 2, 2024 – Dec 3, 2024

lore
Share & tools
lore

matoro

author
#0 lore raw

Hi Helge, when booting 6.12 here myself and another user (CC'd) both observed our ext4 filesystems to be immediately corrupted in the same manner.

Every file that is read or written will have its access/modify times set to 2446-05-10 18:38:55.0000, which is the maximum ext4 timestamp. The 32-bit userspace doesn't seem to be able to handle this at all, as every further stat() call will error with "Value too large for defined data type". Unfortunately, simply rolling back to kernel 6.11 is insufficient to recover, as the filesystem corruption is persistent, and the errors come from userspace attempting to read the modified files. I was able to recover with a command like: find / -newermt 2446-01-01 -o -newerct 2446-01-01 -o -newerat 2446-01-01 | xargs touch -h

Luckily, lindholm was able to bisect and identified as the culprit commit: b5ff52be891347f8847872c49d7a5c2fa29400a7 ("parisc: Convert to generic clockevents"). Some other comments from the discussion:

17:20:37 <awilfox> would be curious if keeping that patch + CONFIG_SMP=n fixes it
17:20:44 <awilfox> this doesn't look necessarily correct on MP machines
17:23:56 <awilfox> time_keeper_id is now unused; the old code specifically marked the clocksource as unstable on MP machines despite having per_cpu before 17:24:11 <awilfox> and now it seems to imply CLOCK_EVT_FEAT_PERCPU is enough to work around it
17:24:13 <awilfox> maybe it isn't

Thanks!

#1 lore raw

I haven't seen any file system corruption on rp3440 with several weeks of running with clock events.  I just started running 6.12.1 today though.

I have the following timer config:

# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y

# end of Timers subsystem

There was some concern about this change on systems where the CPU timers aren't synchronized.  what systems do you see this on?

Dave

Quoted 14 lines · matoro

Hi Helge, when booting 6.12 here myself and another user (CC'd) both observed our ext4 filesystems to be immediately corrupted in the same manner.

Every file that is read or written will have its access/modify times set to 2446-05-10 18:38:55.0000, which is the maximum ext4 timestamp.  The 32-bit userspace doesn't seem to be able to handle this at all, as every further stat() call will error with "Value too large for defined data type".  Unfortunately, simply rolling back to kernel 6.11 is insufficient to recover, as the filesystem corruption is persistent, and the errors come from userspace attempting to read the modified files.  I was able to recover with a command like:  find / -newermt 2446-01-01 -o -newerct 2446-01-01 -o -newerat 2446-01-01 | xargs touch -h

Luckily, lindholm was able to bisect and identified as the culprit commit:  b5ff52be891347f8847872c49d7a5c2fa29400a7 ("parisc: Convert to generic clockevents").  Some other comments from the discussion:

17:20:37 <awilfox> would be curious if keeping that patch + CONFIG_SMP=n fixes it
17:20:44 <awilfox> this doesn't look necessarily correct on MP machines
17:23:56 <awilfox> time_keeper_id is now unused; the old code specifically marked the clocksource as unstable on MP machines despite having per_cpu before 17:24:11 <awilfox> and now it seems to imply CLOCK_EVT_FEAT_PERCPU is enough to work around it
17:24:13 <awilfox> maybe it isn't

Thanks!

Signature · 1 line
John David Anglin  [email protected]
author
#2 lore raw

Hmm, this is my config, also on an rp3440:

#
# Timers subsystem
#
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set

# end of Timers subsystem

lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.

Quoted 34 lines · John David Anglin

I haven't seen any file system corruption on rp3440 with several weeks of running with clock events.  I just started running 6.12.1 today though.

I have the following timer config:

# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y

# end of Timers subsystem

There was some concern about this change on systems where the CPU timers aren't synchronized.  what systems do you see this on?

Dave

Quoted 13 lines · matoro

Hi Helge, when booting 6.12 here myself and another user (CC'd) both observed our ext4 filesystems to be immediately corrupted in the same manner.

Every file that is read or written will have its access/modify times set to 2446-05-10 18:38:55.0000, which is the maximum ext4 timestamp.  The 32-bit userspace doesn't seem to be able to handle this at all, as every further stat() call will error with "Value too large for defined data type".  Unfortunately, simply rolling back to kernel 6.11 is insufficient to recover, as the filesystem corruption is persistent, and the errors come from userspace attempting to read the modified files.  I was able to recover with a command like:  find / -newermt 2446-01-01 -o -newerct 2446-01-01 -o -newerat 2446-01-01 | xargs touch -h

Luckily, lindholm was able to bisect and identified as the culprit commit:  b5ff52be891347f8847872c49d7a5c2fa29400a7 ("parisc: Convert to generic clockevents").  Some other comments from the discussion:

17:20:37 <awilfox> would be curious if keeping that patch + CONFIG_SMP=n fixes it
17:20:44 <awilfox> this doesn't look necessarily correct on MP machines
17:23:56 <awilfox> time_keeper_id is now unused; the old code specifically marked the clocksource as unstable on MP machines despite having per_cpu before 17:24:11 <awilfox> and now it seems to imply CLOCK_EVT_FEAT_PERCPU is enough to work around it
17:24:13 <awilfox> maybe it isn't

Thanks!

#3 lore raw
Quoted 10 lines · matoro

Hmm, this is my config, also on an rp3440:

#
# Timers subsystem
#
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set

Quoted 4 lines
# end of Timers subsystem

lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.

Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel CONFIG_SMP=n will mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)

Magnus Lindholm

#4 lore raw
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro

Hmm, this is my config, also on an rp3440:

#
# Timers subsystem
#
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set

# end of Timers subsystem
Quoted 6 lines
Quoted 2 lines

lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.

Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel CONFIG_SMP=n will mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)

The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details.  I have used it on rp3440 and c8000.

I would try my settings.  The primary reason in switching to the clockevent drivers was to improve clock resolution.  The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests.  If config is the issue, probably CONFIG_HIGH_RES_TIMERS needs to be forced when clockevent drivers are used.

Almost every other system uses the clockevent drivers.  So, there was a risk that parisc would become unsupported.

I wonder if this could be caused by dead RTC battery.  Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.

I run ntp on all my machines.

What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?

Dave

Signature · 1 line
John David Anglin  [email protected]
author
#5 lore raw
Quoted 34 lines · John David Anglin
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro

Hmm, this is my config, also on an rp3440:

#
# Timers subsystem
#
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set

# end of Timers subsystem
Quoted 6 lines
Quoted 2 lines

lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.

Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel CONFIG_SMP=n will mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)

The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details.  I have used it on rp3440 and c8000.

I would try my settings.  The primary reason in switching to the clockevent drivers was to improve clock resolution.  The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests.  If config is the issue, probably CONFIG_HIGH_RES_TIMERS needs to be forced when clockevent drivers are used.

Almost every other system uses the clockevent drivers.  So, there was a risk that parisc would become unsupported.

I wonder if this could be caused by dead RTC battery.  Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.

Quoted 6 lines

I run ntp on all my machines.

What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?

Dave

The files that have bad dates seem to be the ones /init on this system touches at early boot. See the output here: https://paste.matoro.tk/8cq8omg

When booted into the bad kernel, date(1) works and displays the correct time.
I'm using chrony for time syncing as well.

After switching to the config specified above, boot hangs before even getting to userspace with the following output:

[   12.473410] 0000:e0:01.1: ttyS2 at MMIO 0xfffffffff4050038 (irq = 73, base_baud = 115200) is a 16550A
[   12.757386] sym0: <1010-66> rev 0x1 at pci 0000:20:01.0 irq 70
[   12.761419] sym0: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[   12.885367] sym0: SCSI BUS has been reset.
[   12.889389] scsi host0: sym-2.2.3
[   13.053380] sym1: <1010-66> rev 0x1 at pci 0000:20:01.1 irq 71
[   13.055515] sym1: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[   13.165367] sym1: SCSI BUS has been reset.
[   13.169388] scsi host1: sym-2.2.3
[   13.208927] rtc-generic rtc-generic: registered as rtc0
[   13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822)
[   13.281367] NET: Registered PF_INET6 protocol family
[   13.281367] Segment Routing with IPv6
[   13.281367] In-situ OAM (IOAM) with IPv6
[   13.281367] registered taskstats version 1
[   13.281367] Unstable clock detected, switching default tracing clock to "global"
[   13.281367] If you want to keep using the local clock, then add:
[   13.281367]   "trace_clock=local"
[   13.281367] on the kernel command line

At the end there the clock seems to stop progressing forward, as there are several real-time seconds that elapse in between messages with the same timestamp. So I'm completely unable to boot with this config at all.

#6 lore raw

Hi Matoro,

Quoted 48 lines · matoro
Quoted 34 lines · John David Anglin
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro

Hmm, this is my config, also on an rp3440:

#
# Timers subsystem
#
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set

# end of Timers subsystem
Quoted 6 lines
Quoted 2 lines

lindholm can confirm on their hardware/config.  Maybe you can try that and see if you can reproduce?  I will try your config as well.

Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel CONFIG_SMP=n will mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)

The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details.  I have used it on rp3440 and c8000.

I would try my settings.  The primary reason in switching to the clockevent drivers was to improve clock resolution.  The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests.  If config is the issue, probably CONFIG_HIGH_RES_TIMERS needs to be forced when clockevent drivers are used.

Almost every other system uses the clockevent drivers.  So, there was a risk that parisc would become unsupported.

I wonder if this could be caused by dead RTC battery.  Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.

Quoted 6 lines

I run ntp on all my machines.

What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?

Dave

The files that have bad dates seem to be the ones /init on this system touches at early boot.  See the output here:  https://paste.matoro.tk/8cq8omg

When booted into the bad kernel, date(1) works and displays the correct time.  I'm using chrony for time syncing as well.

After switching to the config specified above, boot hangs before even getting to userspace with the following output:

Quoted 15 lines

[   12.473410] 0000:e0:01.1: ttyS2 at MMIO 0xfffffffff4050038 (irq = 73, base_baud = 115200) is a 16550A [   12.757386] sym0: <1010-66> rev 0x1 at pci 0000:20:01.0 irq 70
[   12.761419] sym0: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[   12.885367] sym0: SCSI BUS has been reset.
[   12.889389] scsi host0: sym-2.2.3
[   13.053380] sym1: <1010-66> rev 0x1 at pci 0000:20:01.1 irq 71
[   13.055515] sym1: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[   13.165367] sym1: SCSI BUS has been reset.
[   13.169388] scsi host1: sym-2.2.3
[   13.208927] rtc-generic rtc-generic: registered as rtc0
[   13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822) [   13.281367] NET: Registered PF_INET6 protocol family
[   13.281367] Segment Routing with IPv6
[   13.281367] In-situ OAM (IOAM) with IPv6
[   13.281367] registered taskstats version 1

this message...:

Quoted 4 lines

[   13.281367] Unstable clock detected, switching default tracing clock to "global" [   13.281367] If you want to keep using the local clock, then add:
[   13.281367]   "trace_clock=local"
[   13.281367] on the kernel command line

is very misleading and has nothing to do with the parisc system clock from my commit. You will find this message in all older kernels you had booted as well. It refers to the tracing clock, which isn't used here.

Quoted 4 lines
Quoted 1 line

At the end there the clock seems to stop progressing forward, as

there are several real-time seconds that elapse in between messages with the same timestamp. So I'm completely unable to boot with this config at all.

I think the problems you see are manifold, as it's probably not just triggered by the kernel clock patch itself.
I've seen various issues on debian in the past, as long as we hadn't built glibc with 64-bit time64_t enabled.
One such issue is what you see now: your system fails to bring up userspace because the file date stamps are a lot in the future (even if the dates are wrong, it should be able to boot).

So, the kernel patch might trigger some time inconsistencies, but having non-time64_t enabled glibc is probably the main reason why your boot fails.

Both Dave and myself did test this commit:
commit b5ff52be891347f8847872c49d7a5c2fa29400a7
parisc: Convert to generic clockevents

Nevertheless, maybe on your system one of the SMP CPUs returned an invalid time (e.g. outside of 32-bit range) and thus your glibc seems to have gone crazy. That's currently the only idea I have right now which somehow might explain your situation.

I think we need more testing.

Helge

#7 lore raw
In reply to matoro
Quoted 68 lines · matoro
Quoted 34 lines · John David Anglin
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro

Hmm, this is my config, also on an rp3440:

#
# Timers subsystem
#
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set

# end of Timers subsystem
Quoted 6 lines
Quoted 2 lines

lindholm can confirm on their hardware/config.  Maybe you can try that and see if you can reproduce?  I will try your config as well.

Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel CONFIG_SMP=n will mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)

The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details.  I have used it on rp3440 and c8000.

I would try my settings.  The primary reason in switching to the clockevent drivers was to improve clock resolution.  The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests.  If config is the issue, probably CONFIG_HIGH_RES_TIMERS needs to be forced when clockevent drivers are used.

Almost every other system uses the clockevent drivers.  So, there was a risk that parisc would become unsupported.

I wonder if this could be caused by dead RTC battery.  Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.

Quoted 6 lines

I run ntp on all my machines.

What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?

Dave

The files that have bad dates seem to be the ones /init on this system touches at early boot.  See the output here: https://paste.matoro.tk/8cq8omg

When booted into the bad kernel, date(1) works and displays the correct time.  I'm using chrony for time syncing as well.

After switching to the config specified above, boot hangs before even getting to userspace with the following output:

[   12.473410] 0000:e0:01.1: ttyS2 at MMIO 0xfffffffff4050038 (irq = 73, base_baud = 115200) is a 16550A [   12.757386] sym0: <1010-66> rev 0x1 at pci 0000:20:01.0 irq 70
[   12.761419] sym0: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[   12.885367] sym0: SCSI BUS has been reset.
[   12.889389] scsi host0: sym-2.2.3
[   13.053380] sym1: <1010-66> rev 0x1 at pci 0000:20:01.1 irq 71
[   13.055515] sym1: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[   13.165367] sym1: SCSI BUS has been reset.
[   13.169388] scsi host1: sym-2.2.3
[   13.208927] rtc-generic rtc-generic: registered as rtc0
[   13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822) [   13.281367] NET: Registered PF_INET6 protocol family
[   13.281367] Segment Routing with IPv6
[   13.281367] In-situ OAM (IOAM) with IPv6
[   13.281367] registered taskstats version 1
[   13.281367] Unstable clock detected, switching default tracing clock to "global"
[   13.281367] If you want to keep using the local clock, then add:
[   13.281367]   "trace_clock=local"
[   13.281367] on the kernel command line

Quoted 1 line

At the end there the clock seems to stop progressing forward, as there are several real-time seconds that elapse in between messages with the same timestamp.  So I'm completely unable to boot with this config at all.

I don't see "Unstable clock detected" message.

I also have in config:
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GENERIC_SCHED_CLOCK=y

Clock seems to get stuck here;
[   13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822)

On mx3210, clock continues to increment:
[    1.995462] rtc-generic rtc-generic: registered as rtc0
[    2.003158] rtc-generic rtc-generic: setting system clock to 2024-12-01T15:23:25 UTC (1733066605) [    2.003719] IR JVC protocol handler initialized
[    2.004109] IR MCE Keyboard/mouse protocol handler initialized

Dave

Signature · 1 line
John David Anglin  [email protected]
#8 lore raw
Quoted 2 lines · John David Anglin

I also have in config:
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y

Here is a comment from <https://www.kernel.org/doc/Documentation/timers/timekeeping.txt>:

On SMP systems, it is crucial for performance that sched_clock() can be called independently on each CPU without any synchronization performance hits. Some hardware (such as the x86 TSC) will cause the sched_clock() function to drift between the CPUs on the system. The kernel can work around this by enabling the CONFIG_HAVE_UNSTABLE_SCHED_CLOCK option. This is another aspect that makes sched_clock() different from the ordinary clock source.

Dave

Signature · 1 line
John David Anglin  [email protected]