ext4 corruption on parisc since 6.12
Hi Helge, when booting 6.12 here myself and another user (CC'd) both observed our ext4 filesystems to be immediately corrupted in the same manner.
Every file that is read or written will have its access/modify times set to 2446-05-10 18:38:55.0000, which is the maximum ext4 timestamp. The 32-bit userspace doesn't seem to be able to handle this at all, as every further stat() call will error with "Value too large for defined data type". Unfortunately, simply rolling back to kernel 6.11 is insufficient to recover, as the filesystem corruption is persistent, and the errors come from userspace attempting to read the modified files. I was able to recover with a command like: find / -newermt 2446-01-01 -o -newerct 2446-01-01 -o -newerat 2446-01-01 | xargs touch -h
Luckily, lindholm was able to bisect and identified as the culprit commit: b5ff52be891347f8847872c49d7a5c2fa29400a7 ("parisc: Convert to generic clockevents"). Some other comments from the discussion:
17:20:37 <awilfox> would be curious if keeping that patch + CONFIG_SMP=n fixes it
17:20:44 <awilfox> this doesn't look necessarily correct on MP machines
17:23:56 <awilfox> time_keeper_id is now unused; the old code specifically marked the clocksource as unstable on MP machines despite having per_cpu before 17:24:11 <awilfox> and now it seems to imply CLOCK_EVT_FEAT_PERCPU is enough to work around it
17:24:13 <awilfox> maybe it isn't
Thanks!
I haven't seen any file system corruption on rp3440 with several weeks of running with clock events. I just started running 6.12.1 today though.
I have the following timer config:
# Timers subsystem
#CONFIG_TICK_ONESHOT=yCONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not setCONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ is not setCONFIG_HIGH_RES_TIMERS=y
# end of Timers subsystem
There was some concern about this change on systems where the CPU timers aren't synchronized. what systems do you see this on?
Dave
Quoted 14 lines · matoro
Hi Helge, when booting 6.12 here myself and another user (CC'd) both observed our ext4 filesystems to be immediately corrupted in the same manner.
Every file that is read or written will have its access/modify times set to 2446-05-10 18:38:55.0000, which is the maximum ext4 timestamp. The 32-bit userspace doesn't seem to be able to handle this at all, as every further
stat()call will error with "Value too large for defined data type". Unfortunately, simply rolling back to kernel 6.11 is insufficient to recover, as the filesystem corruption is persistent, and the errors come from userspace attempting to read the modified files. I was able to recover with a command like: find / -newermt 2446-01-01 -o -newerct 2446-01-01 -o -newerat 2446-01-01 | xargs touch -hLuckily, lindholm was able to bisect and identified as the culprit commit:
b5ff52be891347f8847872c49d7a5c2fa29400a7("parisc: Convert to generic clockevents"). Some other comments from the discussion:17:20:37 <awilfox> would be curious if keeping that patch +
CONFIG_SMP=nfixes it
17:20:44 <awilfox> this doesn't look necessarily correct on MP machines
17:23:56 <awilfox>time_keeper_idis now unused; the old code specifically marked the clocksource as unstable on MP machines despite havingper_cpubefore 17:24:11 <awilfox> and now it seems to implyCLOCK_EVT_FEAT_PERCPUis enough to work around it
17:24:13 <awilfox> maybe it isn'tThanks!
Signature · 1 line
John David Anglin [email protected]
Hmm, this is my config, also on an rp3440:
#
# Timers subsystem
#CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set
# end of Timers subsystem
lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.
Quoted 34 lines · John David Anglin
I haven't seen any file system corruption on rp3440 with several weeks of running with clock events. I just started running 6.12.1 today though.
I have the following timer config:
# Timers subsystem
#CONFIG_TICK_ONESHOT=yCONFIG_NO_HZ_COMMON=y
#CONFIG_HZ_PERIODICis not setCONFIG_NO_HZ_IDLE=y
#CONFIG_NO_HZis not setCONFIG_HIGH_RES_TIMERS=y# end of Timers subsystemThere was some concern about this change on systems where the CPU timers aren't synchronized. what systems do you see this on?
Dave
Quoted 13 lines · matoro
Hi Helge, when booting 6.12 here myself and another user (CC'd) both observed our ext4 filesystems to be immediately corrupted in the same manner.
Every file that is read or written will have its access/modify times set to 2446-05-10 18:38:55.0000, which is the maximum ext4 timestamp. The 32-bit userspace doesn't seem to be able to handle this at all, as every further
stat()call will error with "Value too large for defined data type". Unfortunately, simply rolling back to kernel 6.11 is insufficient to recover, as the filesystem corruption is persistent, and the errors come from userspace attempting to read the modified files. I was able to recover with a command like: find / -newermt 2446-01-01 -o -newerct 2446-01-01 -o -newerat 2446-01-01 | xargs touch -hLuckily, lindholm was able to bisect and identified as the culprit commit:
b5ff52be891347f8847872c49d7a5c2fa29400a7("parisc: Convert to generic clockevents"). Some other comments from the discussion:17:20:37 <awilfox> would be curious if keeping that patch +
CONFIG_SMP=nfixes it
17:20:44 <awilfox> this doesn't look necessarily correct on MP machines
17:23:56 <awilfox>time_keeper_idis now unused; the old code specifically marked the clocksource as unstable on MP machines despite havingper_cpubefore 17:24:11 <awilfox> and now it seems to implyCLOCK_EVT_FEAT_PERCPUis enough to work around it
17:24:13 <awilfox> maybe it isn'tThanks!
Quoted 10 lines · matoro
Hmm, this is my config, also on an rp3440:
#
# Timers subsystem
#CONFIG_HZ_PERIODIC=y
#CONFIG_NO_HZ_IDLEis not set
#CONFIG_NO_HZis not set
#CONFIG_HIGH_RES_TIMERSis not set
Quoted 4 lines
# end of Timers subsystemlindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.
Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel CONFIG_SMP=n will mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)
Magnus Lindholm
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro
Hmm, this is my config, also on an rp3440:
#
# Timers subsystem
#CONFIG_HZ_PERIODIC=y
#CONFIG_NO_HZ_IDLEis not set
#CONFIG_NO_HZis not set
#CONFIG_HIGH_RES_TIMERSis not set# end of Timers subsystem
Quoted 6 lines
Quoted 2 lines
lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.
Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel
CONFIG_SMP=nwill mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)
The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details. I have used it on rp3440 and c8000.
I would try my settings. The primary reason in switching to the clockevent drivers was to improve clock resolution. The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests. If config is the issue, probably CONFIG_HIGH_RES_TIMERS needs to be forced when clockevent drivers are used.
Almost every other system uses the clockevent drivers. So, there was a risk that parisc would become unsupported.
I wonder if this could be caused by dead RTC battery. Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.
I run ntp on all my machines.
What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?
Dave
Signature · 1 line
John David Anglin [email protected]
Quoted 34 lines · John David Anglin
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro
Hmm, this is my config, also on an rp3440:
#
# Timers subsystem
#CONFIG_HZ_PERIODIC=y
#CONFIG_NO_HZ_IDLEis not set
#CONFIG_NO_HZis not set
#CONFIG_HIGH_RES_TIMERSis not set# end of Timers subsystemQuoted 6 lines
Quoted 2 lines
lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.
Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel
CONFIG_SMP=nwill mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details. I have used it on rp3440 and c8000.
I would try my settings. The primary reason in switching to the clockevent drivers was to improve clock resolution. The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests. If config is the issue, probably
CONFIG_HIGH_RES_TIMERSneeds to be forced when clockevent drivers are used.Almost every other system uses the clockevent drivers. So, there was a risk that parisc would become unsupported.
I wonder if this could be caused by dead RTC battery. Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.
Quoted 6 lines
I run ntp on all my machines.
What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?
Dave
The files that have bad dates seem to be the ones /init on this system touches at early boot. See the output here: https://paste.matoro.tk/8cq8omg
When booted into the bad kernel, date(1) works and displays the correct time.
I'm using chrony for time syncing as well.
After switching to the config specified above, boot hangs before even getting to userspace with the following output:
[ 12.473410] 0000:e0:01.1: ttyS2 at MMIO 0xfffffffff4050038 (irq = 73, base_baud = 115200) is a 16550A [ 12.757386] sym0: <1010-66> rev 0x1 at pci 0000:20:01.0 irq 70 [ 12.761419] sym0: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking [ 12.885367] sym0: SCSI BUS has been reset. [ 12.889389] scsi host0: sym-2.2.3 [ 13.053380] sym1: <1010-66> rev 0x1 at pci 0000:20:01.1 irq 71 [ 13.055515] sym1: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking [ 13.165367] sym1: SCSI BUS has been reset. [ 13.169388] scsi host1: sym-2.2.3 [ 13.208927] rtc-generic rtc-generic: registered as rtc0 [ 13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822) [ 13.281367] NET: Registered PF_INET6 protocol family [ 13.281367] Segment Routing with IPv6 [ 13.281367] In-situ OAM (IOAM) with IPv6 [ 13.281367] registered taskstats version 1 [ 13.281367] Unstable clock detected, switching default tracing clock to "global" [ 13.281367] If you want to keep using the local clock, then add: [ 13.281367] "trace_clock=local" [ 13.281367] on the kernel command line
At the end there the clock seems to stop progressing forward, as there are several real-time seconds that elapse in between messages with the same timestamp. So I'm completely unable to boot with this config at all.
Hi Matoro,
Quoted 48 lines · matoro
Quoted 34 lines · John David Anglin
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro
Hmm, this is my config, also on an rp3440:
#
# Timers subsystem
#CONFIG_HZ_PERIODIC=y
#CONFIG_NO_HZ_IDLEis not set
#CONFIG_NO_HZis not set
#CONFIG_HIGH_RES_TIMERSis not set# end of Timers subsystemQuoted 6 lines
Quoted 2 lines
lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.
Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel
CONFIG_SMP=nwill mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details. I have used it on rp3440 and c8000.
I would try my settings. The primary reason in switching to the clockevent drivers was to improve clock resolution. The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests. If config is the issue, probably
CONFIG_HIGH_RES_TIMERSneeds to be forced when clockevent drivers are used.Almost every other system uses the clockevent drivers. So, there was a risk that parisc would become unsupported.
I wonder if this could be caused by dead RTC battery. Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.
Quoted 6 lines
I run ntp on all my machines.
What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?
Dave
The files that have bad dates seem to be the ones /init on this system touches at early boot. See the output here: https://paste.matoro.tk/8cq8omg
When booted into the bad kernel, date(1) works and displays the correct time. I'm using chrony for time syncing as well.
After switching to the config specified above, boot hangs before even getting to userspace with the following output:
Quoted 15 lines
[ 12.473410] 0000:e0:01.1: ttyS2 at MMIO
0xfffffffff4050038(irq = 73,base_baud= 115200) is a 16550A [ 12.757386] sym0: <1010-66> rev0x1at pci 0000:20:01.0 irq 70
[ 12.761419] sym0: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[ 12.885367] sym0: SCSI BUS has been reset.
[ 12.889389] scsi host0: sym-2.2.3
[ 13.053380] sym1: <1010-66> rev0x1at pci 0000:20:01.1 irq 71
[ 13.055515] sym1: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[ 13.165367] sym1: SCSI BUS has been reset.
[ 13.169388] scsi host1: sym-2.2.3
[ 13.208927] rtc-generic rtc-generic: registered as rtc0
[ 13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822) [ 13.281367] NET: RegisteredPF_INET6protocol family
[ 13.281367] Segment Routing with IPv6
[ 13.281367] In-situ OAM (IOAM) with IPv6
[ 13.281367] registered taskstats version 1
this message...:
Quoted 4 lines
[ 13.281367] Unstable clock detected, switching default tracing clock to "global" [ 13.281367] If you want to keep using the local clock, then add:
[ 13.281367] "trace_clock=local"
[ 13.281367] on the kernel command line
is very misleading and has nothing to do with the parisc system clock from my commit. You will find this message in all older kernels you had booted as well. It refers to the tracing clock, which isn't used here.
Quoted 4 lines
Quoted 1 line
At the end there the clock seems to stop progressing forward, as
there are several real-time seconds that elapse in between messages with the same timestamp. So I'm completely unable to boot with this config at all.
I think the problems you see are manifold, as it's probably not just triggered by the kernel clock patch itself.
I've seen various issues on debian in the past, as long as we hadn't built glibc with 64-bit time64_t enabled.
One such issue is what you see now: your system fails to bring up userspace because the file date stamps are a lot in the future (even if the dates are wrong, it should be able to boot).
So, the kernel patch might trigger some time inconsistencies, but having non-time64_t enabled glibc is probably the main reason why your boot fails.
Both Dave and myself did test this commit:
commit b5ff52be891347f8847872c49d7a5c2fa29400a7
parisc: Convert to generic clockevents
Nevertheless, maybe on your system one of the SMP CPUs returned an invalid time (e.g. outside of 32-bit range) and thus your glibc seems to have gone crazy. That's currently the only idea I have right now which somehow might explain your situation.
I think we need more testing.
Helge
Quoted 68 lines · matoro
Quoted 34 lines · John David Anglin
Quoted 13 lines · Magnus Lindholm
Quoted 11 lines · matoro
Hmm, this is my config, also on an rp3440:
#
# Timers subsystem
#CONFIG_HZ_PERIODIC=y
#CONFIG_NO_HZ_IDLEis not set
#CONFIG_NO_HZis not set
#CONFIG_HIGH_RES_TIMERSis not set# end of Timers subsystemQuoted 6 lines
Quoted 2 lines
lindholm can confirm on their hardware/config. Maybe you can try that and see if you can reproduce? I will try your config as well.
Hi, I'm on a HPC8000 "parisc64 PA8800 (Mako) 9000/785/C8000". I can confirm that building a kernel
CONFIG_SMP=nwill mitigate this problem.
I haven't messed around with the config in the Timer subsystem so in my case the parameters suggested are unset. (my config looks like matoros)The clockevent driver was tested on both rp3440 and c8000, and some other SMP machines. Helge knows details. I have used it on rp3440 and c8000.
I would try my settings. The primary reason in switching to the clockevent drivers was to improve clock resolution. The best resolution with the old drivers was 1 ms at 1000 HZ. This caused problems with various package tests. If config is the issue, probably
CONFIG_HIGH_RES_TIMERSneeds to be forced when clockevent drivers are used.Almost every other system uses the clockevent drivers. So, there was a risk that parisc would become unsupported.
I wonder if this could be caused by dead RTC battery. Did you check output of date command? Maybe a dead RTC battery interacts badly with clockevent drivers.
Quoted 6 lines
I run ntp on all my machines.
What files have bad dates (i.e., is this really a ext4 file system issue) or is it just that system has a bad clock?
Dave
The files that have bad dates seem to be the ones /init on this system touches at early boot. See the output here: https://paste.matoro.tk/8cq8omg
When booted into the bad kernel, date(1) works and displays the correct time. I'm using chrony for time syncing as well.
After switching to the config specified above, boot hangs before even getting to userspace with the following output:
[ 12.473410] 0000:e0:01.1: ttyS2 at MMIO
0xfffffffff4050038(irq = 73,base_baud= 115200) is a 16550A [ 12.757386] sym0: <1010-66> rev0x1at pci 0000:20:01.0 irq 70
[ 12.761419] sym0: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[ 12.885367] sym0: SCSI BUS has been reset.
[ 12.889389] scsi host0: sym-2.2.3
[ 13.053380] sym1: <1010-66> rev0x1at pci 0000:20:01.1 irq 71
[ 13.055515] sym1: PA-RISC Firmware, ID 7, Fast-80, LVD, parity checking
[ 13.165367] sym1: SCSI BUS has been reset.
[ 13.169388] scsi host1: sym-2.2.3
[ 13.208927] rtc-generic rtc-generic: registered as rtc0
[ 13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822) [ 13.281367] NET: RegisteredPF_INET6protocol family
[ 13.281367] Segment Routing with IPv6
[ 13.281367] In-situ OAM (IOAM) with IPv6
[ 13.281367] registered taskstats version 1
[ 13.281367] Unstable clock detected, switching default tracing clock to "global"
[ 13.281367] If you want to keep using the local clock, then add:
[ 13.281367] "trace_clock=local"
[ 13.281367] on the kernel command line
Quoted 1 line
At the end there the clock seems to stop progressing forward, as there are several real-time seconds that elapse in between messages with the same timestamp. So I'm completely unable to boot with this config at all.
I don't see "Unstable clock detected" message.
I also have in config:CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=yCONFIG_GENERIC_SCHED_CLOCK=y
Clock seems to get stuck here;
[ 13.281367] rtc-generic rtc-generic: setting system clock to 2024-12-02T07:17:02 UTC (1733123822)
On mx3210, clock continues to increment:
[ 1.995462] rtc-generic rtc-generic: registered as rtc0
[ 2.003158] rtc-generic rtc-generic: setting system clock to 2024-12-01T15:23:25 UTC (1733066605) [ 2.003719] IR JVC protocol handler initialized
[ 2.004109] IR MCE Keyboard/mouse protocol handler initialized
Dave
Signature · 1 line
John David Anglin [email protected]
Quoted 2 lines · John David Anglin
I also have in config:
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
Here is a comment from <https://www.kernel.org/doc/Documentation/timers/timekeeping.txt>:
On SMP systems, it is crucial for performance that sched_clock() can be called independently on each CPU without any synchronization performance hits. Some hardware (such as the x86 TSC) will cause the sched_clock() function to drift between the CPUs on the system. The kernel can work around this by enabling the CONFIG_HAVE_UNSTABLE_SCHED_CLOCK option. This is another aspect that makes sched_clock() different from the ordinary clock source.
Dave
Signature · 1 line
John David Anglin [email protected]