Skip to contentSkip to conversation

Boot stall regression from "printk for 5.19" merge

PATCH Superseded Marek Behún 10 participants 26 messages Jun 19, 2022 – Jul 4, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

Hello Linus, Petr, John,

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

The stall example:

...

  [    1.108141] io scheduler mq-deadline registered
  [    1.108143] io scheduler kyber registered
  [    1.163484] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [

Or another:

...

  [    1.108155] io scheduler mq-deadline registered
  [    1.108156] io scheduler kyber registered
  [    1.162991] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [    1.163361] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
  [    1.180827] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
  [  

The garbage example:

...

  [    0.920951] raid6: using neon recovery algorithm
  [    0.921228] iommu: Default domain type: Translated
  %
          gb@k+cFL/[    4.954974] DSA: tree 0 setup
  [    4.955286] cfg80211: Loading compiled-in X.509 certificates for regulatory database

I did not have time to look at the contents of the commit yet to try to fix it myself, and won't have time for another two weeks. But I thought I should let you know now.

Marek

maintainer
#1 lore raw

Hi Marek,

Quoted 5 lines · Marek Behún

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

Quoted 7 lines

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Could you provide your kernel config and a complete kernel boot log (at least as much as you get)? I would like to investigate the serial driver. What you are describing sounds more like a serial bug. The kthread console series has already uncovered serial driver bugs in multiple drivers.

Thanks for reporting this.

John Ogness

maintainer
#2 lore raw
Quoted 6 lines · John Ogness
Quoted 5 lines · Marek Behún

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

Quoted 7 lines
Quoted 7 lines

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Also, have you tried 5.19-rc3? There were several fixes that came in after rc2.

John Ogness

#3 lore raw
Quoted 15 lines · On Mon, 20 Jun 2022 00:37:34 +0206 John Ogness <[email protected]>…
Quoted 6 lines · John Ogness
Quoted 5 lines · Marek Behún

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

Quoted 7 lines
Quoted 7 lines

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Quoted 4 lines

Also, have you tried 5.19-rc3? There were several fixes that came in after rc2.

John Ogness

rc3 still has the issue.

Marek

#4 lore raw
In reply to John Ogness
Quoted 22 lines · On Mon, 20 Jun 2022 00:29:16 +0206 John Ogness <[email protected]>…

Hi Marek,

Quoted 5 lines · Marek Behún

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

Quoted 7 lines

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Could you provide your kernel config and a complete kernel boot log (at least as much as you get)? I would like to investigate the serial driver. What you are describing sounds more like a serial bug. The kthread console series has already uncovered serial driver bugs in multiple drivers.

Quoted 3 lines

Thanks for reporting this.

John Ogness

Hi John,

attaching bootlogs and config.

Marek

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.19.0-rc2-00410-g9776fe0f424b (kabel@dellmb) (aarch64-unknown-linux-gnu-gcc (Gentoo Hardened 10.3.1_p20211126 p0) 10.3.1 20211126, GNU ld (Gentoo 2.37_p1 p2) 2.37) #491 SMP Mon Jun 20 11:00:54 CEST 2022
[    0.000000] Machine model: Marvell Armada CN9130-CRB-B
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000f0512000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x23efd1140-0x23efd2fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000]   node   0: [mem 0x0000000004000000-0x00000000041fffff]
[    0.000000]   node   0: [mem 0x0000000004200000-0x00000000bfffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] cma: Reserved 96 MiB at 0x00000000ba000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS resident on physical CPU 0x0
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 19 pages/cpu s37544 r8192 d32088 u77824
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Fallback order for Node 0: 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2064384
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon=uart8250,mmio32,0xf0512000 root=/dev/mmcblk1p1 rw rootwait pci=pcie_bus_safe cpuidle.off=1 cma=96M oops=panic panic=15
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000b6000000-0x00000000ba000000] (64MB)
[    0.000000] Memory: 8038008K/8388608K available (14528K kernel code, 1232K rwdata, 4472K rodata, 2752K init, 488K bss, 252296K reserved, 98304K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GIC: Adjusting CPU interface base to 0x00000000f022f000
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:160, num:32)
[    0.000000] GICv2m: range[mem 0xf0280000-0xf0280fff], SPI[160:191]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:192, num:32)
[    0.000000] GICv2m: range[mem 0xf0290000-0xf0290fff], SPI[192:223]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:224, num:32)
[    0.000000] GICv2m: range[mem 0xf02a0000-0xf02a0fff], SPI[224:255]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:256, num:32)
[    0.000000] GICv2m: range[mem 0xf02b0000-0xf02b0fff], SPI[256:287]
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[    0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[    0.008184] Console: colour dummy device 80x25
[    0.012710] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.023209] pid_max: default: 32768 minimum: 301
[    0.027926] LSM: Security Framework initializing
[    0.032655] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.040328] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.048990] cblist_init_generic: Setting adjustable number of callback queues.
[    0.056306] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.062565] rcu: Hierarchical SRCU implementation.
[    0.062589] printk: bootconsole [uart8250] printing thread started
[    0.073843] smp: Bringing up secondary CPUs ...
[    0.074238] Detected PIPT I-cache on CPU1
[    0.074267] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.074695] Detected PIPT I-cache on CPU2
[    0.074716] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[    0.101014] Detected PIPT I-cache on CPU3
[    0.101030] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[    0.101061] smp: Brought up 1 node, 4 CPUs
[    0.101064] SMP: Total of 4 processors activated.
[    0.101067] CPU features: detected: 32-bit EL0 Support
[    0.101068] CPU features: detected: CRC32 instructions
[    0.101100] CPU: All CPU(s) started at EL2
[    0.101111] alternatives: patching kernel code
[    0.102096] devtmpfs: initialized
[    0.103487] KASLR enabled
[    0.103528] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.103535] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.104202] pinctrl core: initialized pinctrl subsystem
[    0.104605] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.175823] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.176014] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.176214] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.176235] audit: initializing netlink subsys (disabled)
[    0.176281] audit: type=2000 audit(0.148:1): state=initialized audit_enabled=0 res=1
[    0.176449] thermal_sys: Registered thermal governor 'step_wise'
[    0.176703] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.176742] ASID allocator initialised with 32768 entries
[    0.176745] HugeTLB: can optimize 4095 vmemmap pages for hugepages-1048576kB
[    0.176747] HugeTLB: can optimize 127 vmemmap pages for hugepages-32768kB
[    0.176748] HugeTLB: can optimize 7 vmemmap pages for hugepages-2048kB
[    0.176750] HugeTLB: can optimize 0 vmemmap pages for hugepages-64kB
[    0.176791] Serial: AMBA PL011 UART driver
[    0.264644] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.264648] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.264649] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.264651] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.264864] cryptd: max_cpu_qlen set to 1000
[    0.330807] raid6: neonx8   gen()  6249 MB/s
[    0.398838] raid6: neonx4   gen()  6413 MB/s
[    0.466869] raid6: neonx2   gen()  5630 MB/s
[    0.534902] raid6: neonx1   gen()  4359 MB/s
[    0.602932] raid6: int64x8  gen()  3457 MB/s
[    0.670968] raid6: int64x4  gen()  3741 MB/s
[    0.738998] raid6: int64x2  gen()  3515 MB/s
[    0.807035] raid6: int64x1  gen()  2669 MB/s
[    0.807036] raid6: using algorithm neonx4 gen() 6413 MB/s
[    0.875061] raid6: .... xor() 4669 MB/s, rmw enabled
[    0.875063] raid6: using neon recovery algorithm
[    0.875340] iommu: Default domain type: Translated 
[    0.875342] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.875434] SCSI subsystem initialized
[    0.875547] usbcore: registered new interface driver usbfs
[    0.875557] usbcore: registered new interface driver hub
[    0.875569] usbcore: registered new device driver usb
[    0.875629] usb_phy_generic cp0_usb3_phy0: supply vcc not found, using dummy regulator
[    0.875665] usb_phy_generic cp0_usb3_phy0: dummy supplies not allowed for exclusive requests
[    0.875787] pps_core: LinuxPPS API ver. 1 registered
[    0.875789] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.875793] PTP clock support registered
[    0.876192] vgaarb: loaded
[    0.876296] clocksource: Switched to clocksource arch_sys_counter
[    0.878566] NET: Registered PF_INET protocol family
[    0.878688] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.881003] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.881033] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.881040] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.881227] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.881596] TCP: Hash tables configured (established 65536 bind 65536)
[    0.881640] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.881714] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.881862] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.882100] NET: Registered PF_XDP protocol family
[    0.882105] PCI: CLS 0 bytes, default 64
[    1.044000] Initialise system trusted keyrings
[    1.044085] workingset: timestamp_bits=44 max_order=21 bucket_order=0
[    1.046204] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.046277] fuse: init (API version 7.36)
[    1.064480] NET: Registered PF_ALG protocol family
[    1.064488] xor: measuring software checksum speed
[    1.065414]    8regs           : 10663 MB/sec
[    1.066274]    32regs          : 11465 MB/sec
[    1.067327]    arm64_neon      :  9364 MB/sec
[    1.067328] xor: using function: 32regs (11465 MB/sec)
[    1.067332] Key type asymmetric registered
[    1.067333] Asymmetric key parser 'x509' registered
[    1.067355] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    1.067357] io scheduler mq-deadline registered
[    1.067359] io scheduler kyber registered
[    1.120214] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120577] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137980] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.166562] printk:[ console [ttyS0] printing thread started
 [    1.166564] printk: console [ttyS0] enabled
   1.166486] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 22, base_baud = 12500000) is a 16550A
[    1.166567] printk: bootconsole [uart8250] disabled
[    1.185422] printk: bootconsole [uart8250] printing thread stopped
[    1.188773] brd: module loaded
[    1.190567] loop: module loaded
[    1.190827] megasas: 07.719.03.00-rc1
[    1.191442] ahci f2540000.sata: supply ahci not found, using dummy regulator
[    1.191504] ahci f2540000.sata: supply phy not found, using dummy regulator
[    1.191671] platform f2540000.sata:sata-port@0: supply target not found, using dummy regulator
[    1.192769] spi-nor spi2.0: unrecognized JEDEC id bytes: ff ff ff ff ff ff
[    1.193516] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[    1.193519] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
[    1.193520] MACsec IEEE 802.1AE
[    1.193972] tun: Universal TUN/TAP device driver, 1.6
[    1.194961] hwmon hwmon0: temp1_input not attached to any thermal zone
[    1.198341] mv88e2110 f212a600.mdio-mii:00: Firmware version 8.3.0.0
[    1.198564] hwmon hwmon1: temp1_input not attached to any thermal zone
[    1.284489] mvpp2 f2000000.ethernet: using 8 per-cpu buffers
[    1.290098] rvu_af: Marvell OcteonTX2 RVU Admin Function Driver
[    1.290153] rvu_nicpf: Marvell RVU NIC Physical Function Driver
[    1.290166] rvu_nicvf: Marvell RVU NIC Virtual Function Driver
[    1.290350] usbcore: registered new interface driver r8152
[    1.290363] usbcore: registered new interface driver asix
[    1.290375] usbcore: registered new interface driver ax88179_178a
[    1.290386] usbcore: registered new interface driver cdc_ether
[    1.290397] usbcore: registered new interface driver cdc_eem
[    1.290410] usbcore: registered new interface driver qmi_wwan
[    1.290419] usbcore: registered new interface driver r8153_ecm
[    1.290647] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.290650] ehci-pci: EHCI PCI platform driver
[    1.290661] ehci-platform: EHCI generic platform driver
[    1.290710] ehci-orion: EHCI orion driver
[    1.290753] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.290760] ohci-pci: OHCI PCI platform driver
[    1.290776] ohci-platform: OHCI generic platform driver
[    1.290953] usbcore: registered new interface driver cdc_wdm
[    1.290969] usbcore: registered new interface driver usb-storage
[    1.291001] usbcore: registered new interface driver usbserial_generic
[    1.291008] usbserial: USB Serial support registered for generic
[    1.291018] usbcore: registered new interface driver ftdi_sio
[    1.291025] usbserial: USB Serial support registered for FTDI USB Serial Device
[    1.291034] usbcore: registered new interface driver pl2303
[    1.291039] usbserial: USB Serial support registered for pl2303
[    1.291048] usbcore: registered new interface driver qcserial
[    1.291055] usbserial: USB Serial support registered for Qualcomm USB modem
[    1.291456] armada38x-rtc f2284000.rtc: registered as rtc0
[    1.291468] armada38x-rtc f2284000.rtc: setting system clock to 2047-12-04T05:07:16 UTC (2459048836)
[    1.291545] i2c_dev: i2c /dev entries driver
[    1.307831] sdhci: Secure Digital Host Controller Interface driver
[    1.307834] sdhci: Copyright(c) Pierre Ossman
[    1.307835] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.485145] ledtrig-cpu: registered to indicate activity on CPUs
[    1.485213] usbcore: registered new interface driver usbhid
[    1.485215] usbhid: USB HID core driver
[    1.486546] xt_time: kernel timezone is -0000
[    1.486596] ipip: IPv4 and MPLS over IPv4 tunneling driver
[    1.486773] gre: GRE over IPv4 demultiplexor driver
[    1.486775] ip_gre: GRE over IPv4 tunneling driver
[    1.487150] ipt_CLUSTERIP: ClusterIP Version 0.8 loaded successfully
[    1.487309] NET: Registered PF_INET6 protocol family
[    1.488611] Segment Routing with IPv6
[    1.488614] RPL Segment Routing with IPv6
[    1.488633] In-situ OAM (IOAM) with IPv6
[    1.488670] mip6: Mobile IPv6
[    1.488948] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    1.489255] ip6_gre: GRE over IPv6 tunneling driver
[    1.490153] bpfilter: Loaded bpfilter_umh pid 591
[    1.490511] NET: Registered PF_PACKET protocol family
[    1.490616] 8021q: 802.1Q VLAN Support v1.8
[    1.490639] Key type dns_resolver registered
[    1.490797] registered taskstats version 1
[    1.490806] Loading compiled-in X.509 certificates
[    1.491127] Btrfs loaded, crc32c=crc32c-generic, zoned=no, fsverity=no
[    1.508350] usb_phy_generic cp0_usb3_phy1: dummy supplies not allowed for exclusive requests
[    1.602579] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    1.603807] armada8k-pcie f2600000.pcie: host bridge /cp0/pcie@f2600000 ranges:
[    1.603824] armada8k-pcie f2600000.pcie:      MEM 0x00c0000000..0x00dfefffff -> 0x00c0000000
[    1.603852] armada8k-pcie f2600000.pcie: iATU unroll: disabled
[    1.603854] armada8k-pcie f2600000.pcie: Detected iATU regions: 8 outbound, 8 inbound
[    2.603928] armada8k-pcie f2600000.pcie: Phy link never came up
[    2.604005] armada8k-pcie f2600000.pcie: PCI host bridge to bus 0000:00
[    2.604008] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.604011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xdfefffff]
[    2.604029] pci 0000:00:00.0: [11ab:0110] type 01 class 0x060400
[    2.604037] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[    2.604081] pci 0000:00:00.0: supports D1 D2
[    2.604083] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    2.605111] pci 0000:00:00.0: BAR 0: assigned [mem 0xc0000000-0xc00fffff]
[    2.605118] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    2.605126] pci 0000:00:00.0: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[    3.270938] pcieport 0000:00:00.0: PME: Signaling with IRQ 36
[    3.271062] pcieport 0000:00:00.0: AER: enabled with IRQ 36
[    3.271244] ahci f2540000.sata: supply ahci not found, using dummy regulator
[    3.271297] ahci f2540000.sata: supply phy not found, using dummy regulator
[    3.271333] platform f2540000.sata:sata-port@0: supply target not found, using dummy regulator
[    3.271520] platform f2540000.sata:sata-port@1: supply target not found, using dummy regulator
[    3.272676] ahci f2540000.sata: masking port_map 0x3 -> 0x3
[    3.272710] ahci f2540000.sata: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl platform mode
[    3.272715] ahci f2540000.sata: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs 
[    3.338211] scsi host0: ahci
[    3.338412] scsi host1: ahci
[    3.338480] ata1: SATA max UDMA/133 mmio [mem 0xf2540000-0xf256ffff] port 0x100 irq 37
[    3.338484] ata2: SATA max UDMA/133 mmio [mem 0xf2540000-0xf256ffff] port 0x180 irq 37
[    3.338819] sfp sfp: Host maximum power 3.0W
[    3.340261] mv88e6085 f212a200.mdio-mii:06: switch 0x3930 detected: Marvell 88E6393X, revision 0
[    3.353264] hwmon hwmon2: temp1_input not attached to any thermal zone
[    3.363473] hwmon hwmon3: temp1_input not attached to any thermal zone
[    3.373656] hwmon hwmon4: temp1_input not attached to any thermal zone
[    3.383851] hwmon hwmon5: temp1_input not attached to any thermal zone
[    3.394035] hwmon hwmon6: temp1_input not attached to any thermal zone
[    3.404228] hwmon hwmon7: temp1_input not attached to any thermal zone
[    3.414414] hwmon hwmon8: temp1_input not attached to any thermal zone
[    3.424597] hwmon hwmon9: temp1_input not attached to any thermal zone
[    3.444161] mvpp2 f2000000.ethernet: using 8 per-cpu buffers
[    3.650410] ata2: SATA link down (SStatus 0 SControl 300)
[    3.670329] mvpp2 f2000000.ethernet eth0: Using firmware node mac address 00:0f:b7:06:15:6b
[    3.671958] mvpp2 f2000000.ethernet eth1: Using firmware node mac address 00:0f:b7:06:15:6c
[    3.694323] xhci-hcd f2500000.usb: xHCI Host Controller
[    3.694334] xhci-hcd f2500000.usb: new USB bus registered, assigned bus number 1
[    3.694386] xhci-hcd f2500000.usb: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
[    3.694410] xhci-hcd f2500000.usb: irq 50, io mem 0xf2500000
[    3.694469] xhci-hcd f2500000.usb: xHCI Host Controller
[    3.694474] xhci-hcd f2500000.usb: new USB bus registered, assigned bus number 2
[    3.694478] xhci-hcd f2500000.usb: Host supports USB 3.0 SuperSpeed
[    3.694591] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.19
[    3.694596] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.694598] usb usb1: Product: xHCI Host Controller
[    3.694600] usb usb1: Manufacturer: Linux 5.19.0-rc2-00410-g9776fe0f424b xhci-hcd
[    3.694602] usb usb1: SerialNumber: f2500000.usb
[    3.694802] hub 1-0:1.0: USB hub found
[    3.694824] hub 1-0:1.0: 1 port detected
[    3.694981] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    3.695032] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.19
[    3.695035] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.695037] usb usb2: Product: xHCI Host Controller
[    3.695039] usb usb2: Manufacturer: Linux 5.19.0-rc2-00410-g9776fe0f424b xhci-hcd
[    3.695041] usb usb2: SerialNumber: f2500000.usb
[    3.695193] hub 2-0:1.0: USB hub found
[    3.695209] hub 2-0:1.0: 1 port detected
[    3.706293] xhci-hcd f2510000.usb: xHCI Host Controller
[    3.706302] xhci-hcd f2510000.usb: new USB bus registered, assigned bus number 3
[    3.706359] xhci-hcd f2510000.usb: hcc params 0x0a000990 hci version 0x100 quirks 0x0000000000010010
[    3.706380] xhci-hcd f2510000.usb: irq 51, io mem 0xf2510000
[    3.706440] xhci-hcd f2510000.usb: xHCI Host Controller
[    3.706445] xhci-hcd f2510000.usb: new USB bus registered, assigned bus number 4
[    3.706449] xhci-hcd f2510000.usb: Host supports USB 3.0 SuperSpeed
[    3.706536] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.19
[    3.706541] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.706543] usb usb3: Product: xHCI Host Controller
[    3.706546] usb usb3: Manufacturer: Linux 5.19.0-rc2-00410-g9776fe0f424b xhci-hcd
[    3.706548] usb usb3: SerialNumber: f2510000.usb
[    3.706731] hub 3-0:1.0: USB hub found
[    3.706745] hub 3-0:1.0: 1 port detected
[    3.706884] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    3.706943] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.19
[    3.706946] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.706948] usb usb4: Product: xHCI Host Controller
[    3.706950] usb usb4: Manufacturer: Linux 5.19.0-rc2-00410-g9776fe0f424b xhci-hcd
[    3.706952] usb usb4: SerialNumber: f2510000.usb
[    3.707087] hub 4-0:1.0: USB hub found
[    3.707101] hub 4-0:1.0: 1 port detected
[    3.709114] mv88e6085 f212a200.mdio-mii:06: switch 0x3930 detected: Marvell 88E6393X, revision 0
[    3.721398] hwmon hwmon2: temp1_input not attached to any thermal zone
[    3.730172] hwmon hwmon3: temp1_input not attached to any thermal zone
[    3.738927] hwmon hwmon4: temp1_input not attached to any thermal zone
[    3.739948] mmc0: SDHCI controller on f06e0000.mmc [f06e0000.mmc] using ADMA 64-bit
[    3.747766] hwmon hwmon5: temp1_input not attached to any thermal zone
[    3.756530] hwmon hwmon6: temp1_input not attached to any thermal zone
[    3.765306] hwmon hwmon7: temp1_input not attached to any thermal zone
[    3.772304] ata1: SATA link down (SStatus 0 SControl 300)
[    3.774108] hwmon hwmon8: temp1_input not attached to any thermal zone
[    3.782887] hwmon hwmon9: temp1_input not attached to any thermal zone
[    3.817167] mmc0: mmc_select_hs200 failed, error -74
[    3.819667] xenon-sdhci f06e0000.mmc: Timing issue might occur in DDR mode
[    3.827241] mmc0: new DDR MMC card at address 0001
[    3.827509] mmcblk0: mmc0:0001 P1XXXX 3.60 GiB 
[    3.828487] mmcblk0boot0: mmc0:0001 P1XXXX 2.00 MiB 
[    3.829069] mmcblk0boot1: mmc0:0001 P1XXXX 2.00 MiB 
[    3.829571] mmcblk0rpmb: mmc0:0001 P1XXXX 128 KiB, chardev (237:0)
[    4.663299] mv88e6085 f212a200.mdio-mii:06: configuring for inband/10gbase-r link mode
[    4.734457] mv88e6085 f212a200.mdio-mii:06 p1 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:01] driver [Marvell 88E6393 Family] (irq=65)
[    4.736172] mvpp2 f2000000.ethernet: all ports have a low MTU, switching to per-cpu buffers
[    4.756880] mvpp2 f2000000.ethernet: using 8 per-cpu buffers
[    4.834462] mv88e6085 f212a200.mdio-mii:06 p2 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:02] driver [Marvell 88E6393 Family] (irq=66)
[    4.870440] mv88e6085 f212a200.mdio-mii:06 p3 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:03] driver [Marvell 88E6393 Family] (irq=67)
[    4.938461] mv88e6085 f212a200.mdio-mii:06 p4 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:04] driver [Marvell 88E6393 Family] (irq=68)
[    5.006462] mv88e6085 f212a200.mdio-mii:06 p5 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:05] driver [Marvell 88E6393 Family] (irq=69)
[    5.074463] mv88e6085 f212a200.mdio-mii:06 p6 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:06] driver [Marvell 88E6393 Family] (irq=70)
[    5.142462] mv88e6085 f212a200.mdio-mii:06 p7 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:07] driver [Marvell 88E6393 Family] (irq=71)
[    5.190460] mv88e6085 f212a200.mdio-mii:06 p8 (uninitialized): PHY [!cp0!config-space@f2000000!mdio@12a200!switch0@6!mdio:08] driver [Marvell 88E6393 Family] (irq=72)
[    5.193336] DSA: tree 0 setup
[    5.193651] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    5.193737] xenon-sdhci f2780000.mmc: Got CD GPIO
[    5.194305] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    5.224394] mmc1: SDHCI controller on f2780000.mmc [f2780000.mmc] using ADMA 64-bit
[    5.224491] Waiting for root device /dev/mmcblk1p1...
[    5.290356] mmc1: new ultra high speed SDR104 SDHC card at address aaaa
[    5.290599] mmcblk1: mmc1:aaaa SC32G 29.7 GiB 
[    5.292917]  mmcblk1: p1
[    5.312734] EXT4-fs (mmcblk1p1): mounted filesystem with ordered data mode. Quota mode: disabled.
[    5.312758] VFS: Mounted root (ext4 filesystem) on device 179:97.
[    5.316429] devtmpfs: mounted
[    5.316958] Freeing unused kernel memory: 2752K
[    5.364349] Run /sbin/init as init process
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.19.0-rc2-00410-g9776fe0f424b (kabel@dellmb) (aarch64-unknown-linux-gnu-gcc (Gentoo Hardened 10.3.1_p20211126 p0) 10.3.1 20211126, GNU ld (Gentoo 2.37_p1 p2) 2.37) #491 SMP Mon Jun 20 11:00:54 CEST 2022
[    0.000000] Machine model: Marvell Armada CN9130-CRB-B
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000f0512000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x23efd2140-0x23efd3fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000]   node   0: [mem 0x0000000004000000-0x00000000041fffff]
[    0.000000]   node   0: [mem 0x0000000004200000-0x00000000bfffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] cma: Reserved 96 MiB at 0x00000000ba000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS resident on physical CPU 0x0
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 19 pages/cpu s37544 r8192 d32088 u77824
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Fallback order for Node 0: 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2064384
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon=uart8250,mmio32,0xf0512000 root=/dev/mmcblk1p1 rw rootwait pci=pcie_bus_safe cpuidle.off=1 cma=96M oops=panic panic=15
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000b6000000-0x00000000ba000000] (64MB)
[    0.000000] Memory: 8038016K/8388608K available (14528K kernel code, 1232K rwdata, 4472K rodata, 2752K init, 488K bss, 252288K reserved, 98304K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GIC: Adjusting CPU interface base to 0x00000000f022f000
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:160, num:32)
[    0.000000] GICv2m: range[mem 0xf0280000-0xf0280fff], SPI[160:191]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:192, num:32)
[    0.000000] GICv2m: range[mem 0xf0290000-0xf0290fff], SPI[192:223]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:224, num:32)
[    0.000000] GICv2m: range[mem 0xf02a0000-0xf02a0fff], SPI[224:255]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:256, num:32)
[    0.000000] GICv2m: range[mem 0xf02b0000-0xf02b0fff], SPI[256:287]
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[    0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[    0.008211] Console: colour dummy device 80x25
[    0.012745] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.023244] pid_max: default: 32768 minimum: 301
[    0.027944] LSM: Security Framework initializing
[    0.032728] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.040384] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.049003] cblist_init_generic: Setting adjustable number of callback queues.
[    0.056344] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.062611] rcu: Hierarchical SRCU implementation.
[    0.062635] printk: bootconsole [uart8250] printing thread started
[    0.073966] smp: Bringing up secondary CPUs ...
[    0.074361] Detected PIPT I-cache on CPU1
[    0.074391] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.090125] Detected PIPT I-cache on CPU2
[    0.090147] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[    0.090560] Detected PIPT I-cache on CPU3
[    0.090576] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[    0.090603] smp: Brought up 1 node, 4 CPUs
[    0.090607] SMP: Total of 4 processors activated.
[    0.090609] CPU features: detected: 32-bit EL0 Support
[    0.090610] CPU features: detected: CRC32 instructions
[    0.090642] CPU: All CPU(s) started at EL2
[    0.090652] alternatives: patching kernel code
[    0.091639] devtmpfs: initialized
[    0.092989] KASLR enabled
[    0.093027] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.093034] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.093697] pinctrl core: initialized pinctrl subsystem
[    0.094104] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.177958] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.178150] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.178350] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.178371] audit: initializing netlink subsys (disabled)
[    0.178428] audit: type=2000 audit(0.148:1): state=initialized audit_enabled=0 res=1
[    0.178564] thermal_sys: Registered thermal governor 'step_wise'
[    0.178829] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.227934] ASID allocator initialised with 32768 entries
[    0.227937] HugeTLB: can optimize 4095 vmemmap pages for hugepages-1048576kB
[    0.227939] HugeTLB: can optimize 127 vmemmap pages for hugepages-32768kB
[    0.227941] HugeTLB: can optimize 7 vmemmap pages for hugepages-2048kB
[    0.227943] HugeTLB: can optimize 0 vmemmap pages for hugepages-64kB
[    0.227982] Serial: AMBA PL011 UART driver
[    0.267628] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.267631] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.267633] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.267635] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.267850] cryptd: max_cpu_qlen set to 1000
[    0.334690] raid6: neonx8   gen()  6232 MB/s
[    0.402719] raid6: neonx4   gen()  6388 MB/s
[    0.470749] raid6: neonx2   gen()  5630 MB/s
[    0.538783] raid6: neonx1   gen()  4345 MB/s
[    0.606811] raid6: int64x8  gen()  3457 MB/s
[    0.674843] raid6: int64x4  gen()  3743 MB/s
[    0.742875] raid6: int64x2  gen()  3516 MB/s
[    0.810906] raid6: int64x1  gen()  2669 MB/s
[    0.810908] raid6: using algorithm neonx4 gen() 6388 MB/s
[    0.878934] raid6: .... xor() 4720 MB/s, rmw enabled
[    0.878936] raid6: using neon recovery algorithm
[    0.879221] iommu: Default domain type: Translated 
[    0.879223] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.879309] SCSI subsystem initialized
[    0.879434] usbcore: registered new interface driver usbfs
[    0.879447] usbcore: registered new interface driver hub
[    0.879457] usbcore: registered new device driver usb
[    0.879516] usb_phy_generic cp0_usb3_phy0: supply vcc not found, using dummy regulator
[    0.879555] usb_phy_generic cp0_usb3_phy0: dummy supplies not allowed for exclusive requests
[    0.879677] pps_core: LinuxPPS API ver. 1 registered
[    0.879679] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.879683] PTP clock support registered
[    0.880068] vgaarb: loaded
[    0.880177] clocksource: Switched to clocksource arch_sys_counter
[    0.882432] NET: Registered PF_INET protocol family
[    0.882554] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.884865] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.884895] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.884903] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.885091] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.885459] TCP: Hash tables configured (established 65536 bind 65536)
[    0.885502] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.885576] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.885727] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.886003] NET: Registered PF_XDP protocol family
[    0.886009] PCI: CLS 0 bytes, default 64
[    1.048012] Initialise system trusted keyrings
[    1.048086] workingset: timestamp_bits=44 max_order=21 bucket_order=0
[    1.050213] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.050283] fuse: init (API version 7.36)
[    1.068333] NET: Registered PF_ALG protocol family
[    1.068340] xor: measuring software checksum speed
[    1.069266]    8regs           : 10663 MB/sec
[    1.070124]    32regs          : 11487 MB/sec
[    1.071177]    arm64_neon      :  9364 MB/sec
[    1.071179] xor: using function: 32regs (11487 MB/sec)
[    1.071183] Key type asymmetric registered
[    1.071184] Asymmetric key parser 'x509' registered
[    1.071209] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    1.071212] io scheduler mq-deadline registered
[    1.071214] io scheduler kyber registered
[    1.124272] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.19.0-rc2-00410-g9776fe0f424b (kabel@dellmb) (aarch64-unknown-linux-gnu-gcc (Gentoo Hardened 10.3.1_p20211126 p0) 10.3.1 20211126, GNU ld (Gentoo 2.37_p1 p2) 2.37) #491 SMP Mon Jun 20 11:00:54 CEST 2022
[    0.000000] Machine model: Marvell Armada CN9130-CRB-B
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000f0512000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x23efd2140-0x23efd3fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000]   node   0: [mem 0x0000000004000000-0x00000000041fffff]
[    0.000000]   node   0: [mem 0x0000000004200000-0x00000000bfffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] cma: Reserved 96 MiB at 0x00000000ba000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS resident on physical CPU 0x0
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 19 pages/cpu s37544 r8192 d32088 u77824
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Fallback order for Node 0: 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2064384
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon=uart8250,mmio32,0xf0512000 root=/dev/mmcblk1p1 rw rootwait pci=pcie_bus_safe cpuidle.off=1 cma=96M oops=panic panic=15
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000b6000000-0x00000000ba000000] (64MB)
[    0.000000] Memory: 8038016K/8388608K available (14528K kernel code, 1232K rwdata, 4472K rodata, 2752K init, 488K bss, 252288K reserved, 98304K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GIC: Adjusting CPU interface base to 0x00000000f022f000
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:160, num:32)
[    0.000000] GICv2m: range[mem 0xf0280000-0xf0280fff], SPI[160:191]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:192, num:32)
[    0.000000] GICv2m: range[mem 0xf0290000-0xf0290fff], SPI[192:223]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:224, num:32)
[    0.000000] GICv2m: range[mem 0xf02a0000-0xf02a0fff], SPI[224:255]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:256, num:32)
[    0.000000] GICv2m: range[mem 0xf02b0000-0xf02b0fff], SPI[256:287]
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[    0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[    0.008197] Console: colour dummy device 80x25
[    0.012742] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.023205] pid_max: default: 32768 minimum: 301
[    0.027905] LSM: Security Framework initializing
[    0.032654] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.040345] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.049002] cblist_init_generic: Setting adjustable number of callback queues.
[    0.056309] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.062567] rcu: Hierarchical SRCU implementation.
[    0.062592] printk: bootconsole [uart8250] printing thread started
[    0.073843] smp: Bringing up secondary CPUs ...
[    0.074241] Detected PIPT I-cache on CPU1
[    0.074269] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.074698] Detected PIPT I-cache on CPU2
[    0.074719] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[    0.100948] Detected PIPT I-cache on CPU3
[    0.100963] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[    0.100994] smp: Brought up 1 node, 4 CPUs
[    0.100998] SMP: Total of 4 processors activated.
[    0.101000] CPU features: detected: 32-bit EL0 Support
[    0.101002] CPU features: detected: CRC32 instructions
[    0.101033] CPU: All CPU(s) started at EL2
[    0.101043] alternatives: patching kernel code
[    0.102033] devtmpfs: initialized
[    0.103419] KASLR enabled
[    0.103460] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.103467] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.104134] pinctrl core: initialized pinctrl subsystem
[    0.104540] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.175652] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.175845] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.176044] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.176063] audit: initializing netlink subsys (disabled)
[    0.176110] audit: type=2000 audit(0.148:1): state=initialized audit_enabled=0 res=1
[    0.176274] thermal_sys: Registered thermal governor 'step_wise'
[    0.176527] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.176566] ASID allocator initialised with 32768 entries
[    0.176569] HugeTLB: can optimize 4095 vmemmap pages for hugepages-1048576kB
[    0.176571] HugeTLB: can optimize 127 vmemmap pages for hugepages-32768kB
[    0.176573] HugeTLB: can optimize 7 vmemmap pages for hugepages-2048kB
[    0.176574] HugeTLB: can optimize 0 vmemmap pages for hugepages-64kB
[    0.176615] Serial: AMBA PL011 UART driver
[    0.264562] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.264566] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.264568] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.264570] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.264780] cryptd: max_cpu_qlen set to 1000
[    0.330815] raid6: neonx8   gen()  6241 MB/s
[    0.398847] raid6: neonx4   gen()  6390 MB/s
[    0.466879] raid6: neonx2   gen()  5627 MB/s
[    0.534915] raid6: neonx1   gen()  4346 MB/s
[    0.602947] raid6: int64x8  gen()  3457 MB/s
[    0.670976] raid6: int64x4  gen()  3743 MB/s
[    0.739012] raid6: int64x2  gen()  3517 MB/s
[    0.807044] raid6: int64x1  gen()  2668 MB/s
[    0.807046] raid6: using algorithm neonx4 gen() 6390 MB/s
[    0.875075] raid6: .... xor() 4719 MB/s, rmw enabled
[    0.875077] raid6: using neon recovery algorithm
[    0.875357] iommu: Default domain type: Translated 
[    0.875359] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.875450] SCSI subsystem initialized
[    0.875564] usbcore: registered new interface driver usbfs
[    0.875575] usbcore: registered new interface driver hub
[    0.875586] usbcore: registered new device driver usb
[    0.875647] usb_phy_generic cp0_usb3_phy0: supply vcc not found, using dummy regulator
[    0.875683] usb_phy_generic cp0_usb3_phy0: dummy supplies not allowed for exclusive requests
[    0.875809] pps_core: LinuxPPS API ver. 1 registered
[    0.875810] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.875814] PTP clock support registered
[    0.876217] vgaarb: loaded
[    0.876322] clocksource: Switched to clocksource arch_sys_counter
[    0.878568] NET: Registered PF_INET protocol family
[    0.878689] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.881006] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.881034] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.881042] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.881228] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.881596] TCP: Hash tables configured (established 65536 bind 65536)
[    0.881638] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.881712] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.881857] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.882085] NET: Registered PF_XDP protocol family
[    0.882090] PCI: CLS 0 bytes, default 64
[    1.044253] Initialise system trusted keyrings
[    1.044368] workingset: timestamp_bits=44 max_order=21 bucket_order=0
[    1.046489] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.046566] fuse: init (API version 7.36)
[    1.064436] NET: Registered PF_ALG protocol family
[    1.064443] xor: measuring software checksum speed
[    1.065368]    8regs           : 10663 MB/sec
[    1.066229]    32regs          : 11465 MB/sec
[    1.067282]    arm64_neon      :  9364 MB/sec
[    1.067283] xor: using function: 32regs (11465 MB/sec)
[    1.067287] Key type asymmetric registered
[    1.067288] Asymmetric key parser 'x509' registered
[    1.067310] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    1.067313] io scheduler mq-deadline registered
[    1.067314] io scheduler kyber registered
[    1.120226] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120603] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137975] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138248] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver
[    1.
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.19.0-rc2-00410-g9776fe0f424b (kabel@dellmb) (aarch64-unknown-linux-gnu-gcc (Gentoo Hardened 10.3.1_p20211126 p0) 10.3.1 20211126, GNU ld (Gentoo 2.37_p1 p2) 2.37) #491 SMP Mon Jun 20 11:00:54 CEST 2022
[    0.000000] Machine model: Marvell Armada CN9130-CRB-B
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000f0512000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x23efd2140-0x23efd3fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000]   node   0: [mem 0x0000000004000000-0x00000000041fffff]
[    0.000000]   node   0: [mem 0x0000000004200000-0x00000000bfffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000023fffffff]
[    0.000000] cma: Reserved 96 MiB at 0x00000000ba000000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS resident on physical CPU 0x0
[    0.000000] psci: SMC Calling Convention v1.1
[    0.000000] percpu: Embedded 19 pages/cpu s37544 r8192 d32088 u77824
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Fallback order for Node 0: 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2064384
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: console=ttyS0,115200 earlycon=uart8250,mmio32,0xf0512000 root=/dev/mmcblk1p1 rw rootwait pci=pcie_bus_safe cpuidle.off=1 cma=96M oops=panic panic=15
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000b6000000-0x00000000ba000000] (64MB)
[    0.000000] Memory: 8038016K/8388608K available (14528K kernel code, 1232K rwdata, 4472K rodata, 2752K init, 488K bss, 252288K reserved, 98304K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GIC: Adjusting CPU interface base to 0x00000000f022f000
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GIC: Using split EOI/Deactivate mode
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:160, num:32)
[    0.000000] GICv2m: range[mem 0xf0280000-0xf0280fff], SPI[160:191]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:192, num:32)
[    0.000000] GICv2m: range[mem 0xf0290000-0xf0290fff], SPI[192:223]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:224, num:32)
[    0.000000] GICv2m: range[mem 0xf02a0000-0xf02a0fff], SPI[224:255]
[    0.000000] GICv2m: DT overriding V2M MSI_TYPER (base:256, num:32)
[    0.000000] GICv2m: range[mem 0xf02b0000-0xf02b0fff], SPI[256:287]
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[    0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[    0.008220] Console: colour dummy device 80x25
[    0.012710] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.023147] pid_max: default: 32768 minimum: 301
[    0.027839] LSM: Security Framework initializing
[    0.032571] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.040193] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.048796] cblist_init_generic: Setting adjustable number of callback queues.
[    0.056182] cblist_init_generic: Setting shift to 2 and lim to 1.
[    0.062432] rcu: Hierarchical SRCU implementation.
[    0.062457] printk: bootconsole [uart8250] printing thread started
[    0.073711] smp: Bringing up secondary CPUs ...
[    0.074107] Detected PIPT I-cache on CPU1
[    0.074136] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.074561] Detected PIPT I-cache on CPU2
[    0.074583] CPU2: Booted secondary processor 0x0000000100 [0x410fd083]
[    0.100829] Detected PIPT I-cache on CPU3
[    0.100844] CPU3: Booted secondary processor 0x0000000101 [0x410fd083]
[    0.100876] smp: Brought up 1 node, 4 CPUs
[    0.100879] SMP: Total of 4 processors activated.
[    0.100881] CPU features: detected: 32-bit EL0 Support
[    0.100883] CPU features: detected: CRC32 instructions
[    0.100916] CPU: All CPU(s) started at EL2
[    0.100926] alternatives: patching kernel code
[    0.101912] devtmpfs: initialized
[    0.103320] KASLR enabled
[    0.103361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.103367] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.104031] pinctrl core: initialized pinctrl subsystem
[    0.104435] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.175655] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    0.175846] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.176046] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.176067] audit: initializing netlink subsys (disabled)
[    0.176113] audit: type=2000 audit(0.148:1): state=initialized audit_enabled=0 res=1
[    0.176277] thermal_sys: Registered thermal governor 'step_wise'
[    0.176530] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.176569] ASID allocator initialised with 32768 entries
[    0.176572] HugeTLB: can optimize 4095 vmemmap pages for hugepages-1048576kB
[    0.176574] HugeTLB: can optimize 127 vmemmap pages for hugepages-32768kB
[    0.176576] HugeTLB: can optimize 7 vmemmap pages for hugepages-2048kB
[    0.176577] HugeTLB: can optimize 0 vmemmap pages for hugepages-64kB
[    0.176618] Serial: AMBA PL011 UART driver
[    0.264588] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.264591] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.264593] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.264595] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.264805] cryptd: max_cpu_qlen set to 1000
[    0.330674] raid6: neonx8   gen()  6239 MB/s
[    0.398706] raid6: neonx4   gen()  6385 MB/s
[    0.466739] raid6: neonx2   gen()  5626 MB/s
[    0.534771] raid6: neonx1   gen()  4341 MB/s
[    0.602805] raid6: int64x8  gen()  3457 MB/s
[    0.670836] raid6: int64x4  gen()  3743 MB/s
[    0.738870] raid6: int64x2  gen()  3518 MB/s
[    0.806909] raid6: int64x1  gen()  2670 MB/s
[    0.806910] raid6: using algorithm neonx4 gen() 6385 MB/s
[    0.874935] raid6: .... xor() 4714 MB/s, rmw enabled
[    0.874937] raid6: using neon recovery algorithm
[    0.875213] iommu: Default domain type: Translated 
[    0.875215] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.875305] SCSI subsystem initialized
[    0.875419] usbcore: registered new interface driver usbfs
[    0.875430] usbcore: registered new interface driver hub
[    0.875442] usbcore: registered new device driver usb
[    0.875502] usb_phy_generic cp0_usb3_phy0: supply vcc not found, using dummy regulator
[    0.875538] usb_phy_generic cp0_usb3_phy0: dummy supplies not allowed for exclusive requests
[    0.875660] pps_core: LinuxPPS API ver. 1 registered
[    0.875661] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.875666] PTP clock support registered
[    0.876066] vgaarb: loaded
[    0.876172] clocksource: Switched to clocksource arch_sys_counter
[    0.878422] NET: Registered PF_INET protocol family
[    0.878542] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.880860] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.880889] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.880896] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.881082] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.881451] TCP: Hash tables configured (established 65536 bind 65536)
[    0.881495] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.881570] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)
[    0.881715] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.881941] NET: Registered PF_XDP protocol family
[    0.881946] PCI: CLS 0 bytes, default 64
[    1.044028] Initialise system trusted keyrings
[    1.044182] workingset: timestamp_bits=44 max_order=21 bucket_order=0
[    1.046287] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.046359] fuse: init (API version 7.36)
[    1.064339] NET: Registered PF_ALG protocol family
[    1.064346] xor: measuring software checksum speed
[    1.065271]    8regs           : 10662 MB/sec
[    1.066130]    32regs          : 11487 MB/sec
[    1.067183]    arm64_neon      :  9361 MB/sec
[    1.067185] xor: using function: 32regs (11487 MB/sec)
[    1.067189] Key type asymmetric registered
[    1.067190] Asymmetric key parser 'x509' registered
[    1.067210] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    1.067212] io scheduler mq-deadline registered
[    1.067214] io scheduler kyber registered
[    1.120098] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120466] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137871] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138160] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver

[

maintainer
#5 lore raw
Quoted 9 lines · Marek Behún
Quoted 5 lines · On Mon, 20 Jun 2022 00:29:16 +0206 John Ogness <[email protected]>…
Quoted 4 lines · Marek Behún

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

attaching bootlogs and config.

This is the log when the system booted:

Quoted 5 lines
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.19.0-rc2-00410-g9776fe0f424b (kabel@dellmb) (aarch64-unknown-linux-gnu-gcc (Gentoo Hardened 10.3.1_p20211126 p0) 10.3.1 20211126, GNU ld (Gentoo 2.37_p1 p2) 2.37) #491 SMP Mon Jun 20 11:00:54 CEST 2022
[    0.000000] Machine model: Marvell Armada CN9130-CRB-B
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000f0512000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled

Early console enabled.

Quoted 1 line
[    0.000000] NUMA: No NUMA configuration found

[...]

Quoted 2 lines
[    0.062565] rcu: Hierarchical SRCU implementation.
[    0.062589] printk: bootconsole [uart8250] printing thread started

The early console started being handled by the kthread.

Quoted 2 lines
[    0.073843] smp: Bringing up secondary CPUs ...
[    0.074238] Detected PIPT I-cache on CPU1

[...]

Quoted 6 lines
[    1.067359] io scheduler kyber registered
[    1.120214] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120577] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137980] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.166562] printk:[ console [ttyS0] printing thread started
 [    1.166564] printk: console [ttyS0] enabled

2nd console was added using the properly initialized serial port. It should use the same physical port as the early console.

Both early console and proper console driver has its own kthread.

Quoted 1 line
   1.166486] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 22, base_baud = 12500000) is a 16550A

The line is malformed. I wonder if both early console and proper console used the same port in parallel.

Quoted 2 lines
[    1.166567] printk: bootconsole [uart8250] disabled
[    1.185422] printk: bootconsole [uart8250] printing thread stopped

The early console was disabled. Only the properly initialized serial console is used. All should be fine now.

Quoted 2 lines
[    1.188773] brd: module loaded
[    1.190567] loop: module loaded

[...]

Quoted 2 lines
[    5.316958] Freeing unused kernel memory: 2752K
[    5.364349] Run /sbin/init as init process

And I did not catch any further problem.

So, it looks like that con->write() code is not correctly serialized between the early and normal console.

Now, let's see the last lines of failing logs:

Quoted 9 lines
[    1.071214] io scheduler kyber registered
[    1.124272] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver

[

[    1.067314] io scheduler kyber registered
[    1.120226] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120603] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137975] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138248] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver
Quoted 8 lines

[ 1.

[    1.067214] io scheduler kyber registered
[    1.120098] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120466] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137871] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138160] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver

[

All three logs end in the middle of a line. If you compare it with the "working" log then the end 1-3 lines before the normal console was added.

The console output might is delayed because of the threads. Most likely, the output ended when both early and normal console driver started to use the same port.

I am going to check the driver...

Best Regards,
Petr

maintainer
#6 lore raw

Resending with Ilpo, driver maintainer, in CC.

Quoted 99 lines · Petr Mladek
Quoted 9 lines · Marek Behún
Quoted 5 lines · On Mon, 20 Jun 2022 00:29:16 +0206 John Ogness <[email protected]>…
Quoted 4 lines · Marek Behún

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

attaching bootlogs and config.

This is the log when the system booted:

Quoted 5 lines
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.19.0-rc2-00410-g9776fe0f424b (kabel@dellmb) (aarch64-unknown-linux-gnu-gcc (Gentoo Hardened 10.3.1_p20211126 p0) 10.3.1 20211126, GNU ld (Gentoo 2.37_p1 p2) 2.37) #491 SMP Mon Jun 20 11:00:54 CEST 2022
[    0.000000] Machine model: Marvell Armada CN9130-CRB-B
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000f0512000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled

Early console enabled.

Quoted 1 line
[    0.000000] NUMA: No NUMA configuration found

[...]

Quoted 2 lines
[    0.062565] rcu: Hierarchical SRCU implementation.
[    0.062589] printk: bootconsole [uart8250] printing thread started

The early console started being handled by the kthread.

Quoted 2 lines
[    0.073843] smp: Bringing up secondary CPUs ...
[    0.074238] Detected PIPT I-cache on CPU1

[...]

Quoted 6 lines
[    1.067359] io scheduler kyber registered
[    1.120214] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120577] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137980] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.166562] printk:[ console [ttyS0] printing thread started
 [    1.166564] printk: console [ttyS0] enabled

2nd console was added using the properly initialized serial port. It should use the same physical port as the early console.

Both early console and proper console driver has its own kthread.

Quoted 1 line
   1.166486] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 22, base_baud = 12500000) is a 16550A

The line is malformed. I wonder if both early console and proper console used the same port in parallel.

Quoted 2 lines
[    1.166567] printk: bootconsole [uart8250] disabled
[    1.185422] printk: bootconsole [uart8250] printing thread stopped

The early console was disabled. Only the properly initialized serial console is used. All should be fine now.

Quoted 2 lines
[    1.188773] brd: module loaded
[    1.190567] loop: module loaded

[...]

Quoted 2 lines
[    5.316958] Freeing unused kernel memory: 2752K
[    5.364349] Run /sbin/init as init process

And I did not catch any further problem.

So, it looks like that con->write() code is not correctly serialized between the early and normal console.

Now, let's see the last lines of failing logs:

Quoted 9 lines
[    1.071214] io scheduler kyber registered
[    1.124272] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver

[

[    1.067314] io scheduler kyber registered
[    1.120226] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120603] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137975] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138248] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver
Quoted 8 lines

[ 1.

[    1.067214] io scheduler kyber registered
[    1.120098] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120466] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137871] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138160] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver

[

All three logs end in the middle of a line. If you compare it with the "working" log then the end 1-3 lines before the normal console was added.

The console output might is delayed because of the threads. Most likely, the output ended when both early and normal console driver started to use the same port.

I am going to check the driver...

Best Regards,
Petr

#7 lore raw
Quoted 99 lines · Petr Mladek

Resending with Ilpo, driver maintainer, in CC.

Quoted 96 lines · Petr Mladek
Quoted 9 lines · Marek Behún
Quoted 5 lines · On Mon, 20 Jun 2022 00:29:16 +0206 John Ogness <[email protected]>…
Quoted 4 lines · Marek Behún

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

attaching bootlogs and config.

This is the log when the system booted:

Quoted 5 lines
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.19.0-rc2-00410-g9776fe0f424b (kabel@dellmb) (aarch64-unknown-linux-gnu-gcc (Gentoo Hardened 10.3.1_p20211126 p0) 10.3.1 20211126, GNU ld (Gentoo 2.37_p1 p2) 2.37) #491 SMP Mon Jun 20 11:00:54 CEST 2022
[    0.000000] Machine model: Marvell Armada CN9130-CRB-B
[    0.000000] earlycon: uart8250 at MMIO32 0x00000000f0512000 (options '')
[    0.000000] printk: bootconsole [uart8250] enabled

Early console enabled.

Quoted 1 line
[    0.000000] NUMA: No NUMA configuration found

[...]

Quoted 2 lines
[    0.062565] rcu: Hierarchical SRCU implementation.
[    0.062589] printk: bootconsole [uart8250] printing thread started

The early console started being handled by the kthread.

Quoted 2 lines
[    0.073843] smp: Bringing up secondary CPUs ...
[    0.074238] Detected PIPT I-cache on CPU1

[...]

Quoted 6 lines
[    1.067359] io scheduler kyber registered
[    1.120214] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120577] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137980] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.166562] printk:[ console [ttyS0] printing thread started
 [    1.166564] printk: console [ttyS0] enabled

2nd console was added using the properly initialized serial port. It should use the same physical port as the early console.

Both early console and proper console driver has its own kthread.

Quoted 1 line
   1.166486] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 22, base_baud = 12500000) is a 16550A

The line is malformed. I wonder if both early console and proper console used the same port in parallel.

Quoted 2 lines
[    1.166567] printk: bootconsole [uart8250] disabled
[    1.185422] printk: bootconsole [uart8250] printing thread stopped

The early console was disabled. Only the properly initialized serial console is used. All should be fine now.

Quoted 2 lines
[    1.188773] brd: module loaded
[    1.190567] loop: module loaded

[...]

Quoted 2 lines
[    5.316958] Freeing unused kernel memory: 2752K
[    5.364349] Run /sbin/init as init process

And I did not catch any further problem.

So, it looks like that con->write() code is not correctly serialized between the early and normal console.

Now, let's see the last lines of failing logs:

Quoted 9 lines
[    1.071214] io scheduler kyber registered
[    1.124272] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver

[

[    1.067314] io scheduler kyber registered
[    1.120226] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120603] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137975] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138248] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver
Quoted 8 lines

[ 1.

[    1.067214] io scheduler kyber registered
[    1.120098] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
[    1.120466] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
[    1.137871] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
[    1.138160] mv_xor_v2 f0420000.xor: Marvell Version 2 XOR driver

[

All three logs end in the middle of a line. If you compare it with the "working" log then the end 1-3 lines before the normal console was added.

The console output might is delayed because of the threads. Most likely, the output ended when both early and normal console driver started to use the same port.

I am going to check the driver...

Perhaps try reverting 8f3631f0f6eb ("serial/8250: Use fifo in 8250 console driver") if there's yet another unexplored corner hit.

Signature · 1 line
 i.
#8 lore raw
In reply to Petr Mladek
Quoted 7 lines · Petr Mladek

Both early console and proper console driver has its own kthread.

Quoted 1 line
   1.166486] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 22, base_baud = 12500000) is a 16550A

The line is malformed. I wonder if both early console and proper console used the same port in parallel.

Honestly, I get the feeling that we need to just revert the whole "console from thread" thing.

Because:

Quoted 4 lines

So, it looks like that con->write() code is not correctly serialized between the early and normal console.
[ ... ]
I am going to check the driver...

We really cannot be in the situation that some random driver that used to work no longer does, and causes oopses and/or memory corruption just because it's now entered differently from how it traditionally has been.

The traditional console write code has always been very careful to get exclusive access, and it sounds like that is just plain broken now.

So I don't think this is a "driver is broken". Even if you find a fix for it that makes that particular configuration happy, it sounds like there is a much more fundamental issue at hand: the new printk code just doesn't work well, and does things that are against the way console printing has always worked in the past.

I realize that people have wanted to do printing from proper thread context for ages and ages.

But maybe people should also just admit that "I wanted this" was not "this was actually a good idea".

Linus

maintainer
#9 lore raw
Quoted 24 lines · Linus Torvalds
Quoted 7 lines · Petr Mladek

Both early console and proper console driver has its own kthread.

Quoted 1 line
   1.166486] f0512000.serial: ttyS0 at MMIO 0xf0512000 (irq = 22, base_baud = 12500000) is a 16550A

The line is malformed. I wonder if both early console and proper console used the same port in parallel.

Honestly, I get the feeling that we need to just revert the whole "console from thread" thing.

Because:

Quoted 4 lines

So, it looks like that con->write() code is not correctly serialized between the early and normal console.
[ ... ]
I am going to check the driver...

We really cannot be in the situation that some random driver that used to work no longer does, and causes oopses and/or memory corruption just because it's now entered differently from how it traditionally has been.

Quoted 4 lines

The traditional console write code has always been very careful to get exclusive access, and it sounds like that is just plain broken now.

So I don't think this is a "driver is broken".

I see what you think. There might be so many problems with the drivers because they were never used this way. It looks like we opened a can of worms. It is even more problematic because it causes silent boot crashes and it is hard to debug.

I kind of agree with this and I have started looking at some more generic solution.

All these boot crashes were in exactly the same situation when the proper console was initialized and registered while there was the early console used at the same time. It is a problem because they use the same port.

The parallel use of different consoles should be much more safe because they are much more independent.

There are the following possibilities:

1. Enable the kthreads later after the early consoles are gone. This is easy and should fix all known boot problems.

2. Temporary stop the kthreads and use direct printing when the proper consoles are registered. Well, this might be more complicated because the port might be accessed also before register_console() is called.

3. Another solution would be to use the global conosle_lock() also to synchronize the kthreads against each other. But it would be unfortunate.

I am going to prepare 1st solution.

Best Regards,
Petr

maintainer
#10 lore raw

PATCH 1/1 printk/console: Enable console kthreads only when there is no boot console left

The console kthreads uncovered several races in console drivers. All problems were in situation when a console was being properly initialized and registered while an early console, using the same port, was being used.

These problems are pretty hard to debug because they often result into silent boot crashes. It would be nice to fix them but it looks like a can of worms.

Prevent these problems by delaying the use of console kthreads after all early consoles are gone. It might later be optimized. But let's close this can of worms with a big hammer for now so that they do not break first impression on the kthreads that solve other real problems.

Reported=by: Marek Behún <[email protected]>

---
 kernel/printk/printk.c | 38 +++++++++++++++++++++++++-------------
 1 file changed, 25 insertions(+), 13 deletions(-)
kernel/printk/printk.c +25 -13
index b095fb5f5f61..c0c5e2b6b91d 100644
@@ -3551,6 +3551,19 @@ void __init console_init(void)
 	}
 }
 
+static int __init printk_activate_kthreads(void)
+{
+	struct console *con;
+
+	console_lock();
+	printk_kthreads_available = true;
+	for_each_console(con)
+		printk_start_kthread(con);
+	console_unlock();
+
+	return 0;
+}
+
 /*
  * Some boot consoles access data that is in the init section and which will
  * be discarded after the initcalls have been run. To make sure that no code
@@ -3567,6 +3580,7 @@ void __init console_init(void)
  */
 static int __init printk_late_init(void)
 {
+	bool no_bootcon = true;
 	struct console *con;
 	int ret;
 
@@ -3588,7 +3602,10 @@ static int __init printk_late_init(void)
 			pr_warn("bootconsole [%s%d] uses init memory and must be disabled even before the real one is ready\n",
 				con->name, con->index);
 			unregister_console(con);
+			continue;
 		}
+
+		no_bootcon = false;
 	}
 	ret = cpuhp_setup_state_nocalls(CPUHP_PRINTK_DEAD, "printk:dead", NULL,
 					console_cpu_notify);
@@ -3597,23 +3614,18 @@ static int __init printk_late_init(void)
 					console_cpu_notify, NULL);
 	WARN_ON(ret < 0);
 	printk_sysctl_init();
-	return 0;
-}
-late_initcall(printk_late_init);
-
-static int __init printk_activate_kthreads(void)
-{
-	struct console *con;
 
-	console_lock();
-	printk_kthreads_available = true;
-	for_each_console(con)
-		printk_start_kthread(con);
-	console_unlock();
+	/*
+	 * Some console drivers are not ready to use the same port with
+	 * boot (early) and normal console in parallel. Stay on the safe
+	 * side and enable kthreads only when there is no boot console.
+	 */
+	if (no_bootcon)
+		printk_activate_kthreads();
 
 	return 0;
 }
-early_initcall(printk_activate_kthreads);
+late_initcall(printk_late_init);
 
 #if defined CONFIG_PRINTK
 /* If @con is specified, only wait for that console. Otherwise wait for all. */
Signature · 1 line
2.35.3
maintainer
#11 lore raw

Re: [PATCH] printk/console: Enable console kthreads only when there is no boot console left

Quoted 9 lines · Petr Mladek

The console kthreads uncovered several races in console drivers. All problems were in situation when a console was being properly initialized and registered while an early console, using the same port, was being used.

These problems are pretty hard to debug because they often result into silent boot crashes. It would be nice to fix them but it looks like a can of worms.

Quoted 5 lines

Prevent these problems by delaying the use of console kthreads after all early consoles are gone. It might later be optimized. But let's close this can of worms with a big hammer for now so that they do not break first impression on the kthreads that solve other real problems.

Marek, could you please try if this patch solves the problem for you?

Best Regards,
Petr

author Tested-by on v1 1/1
#12 lore raw

Re: [PATCH] printk/console: Enable console kthreads only when there is no boot console left

In reply to Petr Mladek
Quoted 15 lines · On Mon, 20 Jun 2022 17:14:27 +0200 Petr Mladek <[email protected]> wrote:

The console kthreads uncovered several races in console drivers. All problems were in situation when a console was being properly initialized and registered while an early console, using the same port, was being used.

These problems are pretty hard to debug because they often result into silent boot crashes. It would be nice to fix them but it looks like a can of worms.

Prevent these problems by delaying the use of console kthreads after all early consoles are gone. It might later be optimized. But let's close this can of worms with a big hammer for now so that they do not break first impression on the kthreads that solve other real problems.

Quoted 4 lines

Reported=by: Marek Behún <[email protected]>

This solves the issue. You have a typo in the reported-by tag ('=' instead of '-').

#13 lore raw

Re: [PATCH] printk/console: Enable console kthreads only when there is no boot console left

In reply to Petr Mladek
Quoted 2 lines · Petr Mladek

The console kthreads uncovered several races in console drivers.

I really want to make it clear that this was NOT some kind of "races in drivers".

Console drivers may very well have intentionally avoided taking locks for console output, since the printk output was supposed to be serialized by printk.

Don't try to make this some kind of "buggy drivers" thing. This is on printk, not on anything else.

Assuming this solves all issues, I'm ok with this approach, but I really want this to be clearly about printk being buggy, no "blame the drivers" garbage.

And if there are other issues coming up, we revert the whole thing entirely.

Because printk is too important to play games with, and too important to try to blame drivers.

Linus

maintainer
#14 lore raw

Re: [PATCH] printk/console: Enable console kthreads only when there is no boot console left

Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Petr Mladek

The console kthreads uncovered several races in console drivers.

I really want to make it clear that this was NOT some kind of "races in drivers".

Quoted 6 lines

Console drivers may very well have intentionally avoided taking locks for console output, since the printk output was supposed to be serialized by printk.

Don't try to make this some kind of "buggy drivers" thing. This is on printk, not on anything else.

OK, I see that uart_console_write() is used by early_serial8250_write() without port->lock. It means that it is racy against serial8250_console_write(). It might
cause problems reported by this thread. And you are right that it has never been used in parallel before the kthreads.

But I believe that it might cause real problems. serial8250_console_write() takes port->lock to get serialized against other operations on the port. And there might be some when the same port is added as
a proper serial console.

Today I found that probe_baud() is called from serial8250_console_setup() without port->lock. It does reads and writes. I believe that it might break with the earlycon.

Also the commit 589f892ac8ef244e47c5a ("serial: meson:
acquire port->lock in startup()") fixes a race between meson_serial_port_write() and meson_uart_startup(), where meson_serial_port_write() is used by both early and proper console driver. The problem was there even without kthreads. They just made it more visible.

My colleagues familiar with ARM told me that they heard about boot freezes with early consoles before threads. The kthreads allow to reproduce and fix them. In the end, they make the early consoles more reliable.

Quoted 8 lines

Assuming this solves all issues, I'm ok with this approach, but I really want this to be clearly about printk being buggy, no "blame the drivers" garbage.

And if there are other issues coming up, we revert the whole thing entirely.

Because printk is too important to play games with, and too important to try to blame drivers.

I take printk() really seriously. And I definitely do not want to wave out problems as others problem.

I do not want to release 5.19 with broken printk(). But the kthreads solve real bugs where printk() put the system into knees. I want to invest much more time on improving them and fixing related problems. Unfortunately, linux-next was not able to catch the recently reported problems and we were not able to fix them in advance.

All the recent fixes were generic and should make printk() with kthreads much more reliable. I can't be sure if it will be enough. I could only say that I am going to fix any new ones.

Of course, if people continue reporting problems, we would need to revert it for 5.19. But I would really like to give it another chance later.

Best Regards,
Petr

maintainer
#15 lore raw

Re: [PATCH] printk/console: Enable console kthreads only when there is no boot console left

In reply to Petr Mladek
Quoted 9 lines · Petr Mladek

The console kthreads uncovered several races in console drivers. All problems were in situation when a console was being properly initialized and registered while an early console, using the same port, was being used.

These problems are pretty hard to debug because they often result into silent boot crashes. It would be nice to fix them but it looks like a can of worms.

Quoted 5 lines

Prevent these problems by delaying the use of console kthreads after all early consoles are gone. It might later be optimized. But let's close this can of worms with a big hammer for now so that they do not break first impression on the kthreads that solve other real problems.

The problem is that there are multiple consoles accessing the same hardware. The threaded console printing model did not consider this. The commit message should clearly state the actual problem.

Since early printing does not use the kernel driver model, I do not see a clear solution at this point. But since using early printing is not a typical use, I am OK with this patch.

For the commit message I would suggest something like:

Threaded console printing does not take into consideration that boot consoles may be accessing the same hardware as normal consoles and thus must not be called in parallel.

Since it is currently not possible to identify which consoles are accessing the same hardware, delay threaded console printing activation until it is known that there are no boot consoles registered.

Quoted 16 lines
kernel/printk/printk.c +0 -7
index b095fb5f5f61..c0c5e2b6b91d 100644
@@ -3597,23 +3614,18 @@ static int __init printk_late_init(void)
 					console_cpu_notify, NULL);
 	WARN_ON(ret < 0);
 	printk_sysctl_init();
-	return 0;
-}
-late_initcall(printk_late_init);
-
-static int __init printk_activate_kthreads(void)
-{
-	struct console *con;
 

kernel/printk/printk.c static int __init printk_late_init(void) · line 3622

Quoted 10 lines
kernel/printk/printk.c +5 -5
-	console_lock();
-	printk_kthreads_available = true;
-	for_each_console(con)
-		printk_start_kthread(con);
-	console_unlock();
+	/*
+	 * Some console drivers are not ready to use the same port with
+	 * boot (early) and normal console in parallel. Stay on the safe
+	 * side and enable kthreads only when there is no boot console.
+	 */

As with the commit message, this comment is not pointing out the real issue. I suggest:

/*
* Boot consoles may be accessing the same hardware as normal
* consoles and thus must not be called in parallel. Therefore
* only activate threaded console printing if it is known that
* there are no boot consoles registered.
*/

kernel/printk/printk.c static int __init printk_late_init(void) · line 3631

Quoted 10 lines
kernel/printk/printk.c +3 -1
+	if (no_bootcon)
+		printk_activate_kthreads();
 
 	return 0;
 }
-early_initcall(printk_activate_kthreads);
+late_initcall(printk_late_init);
 
 #if defined CONFIG_PRINTK
 /* If @con is specified, only wait for that console. Otherwise wait for all. */

John Ogness

#16 lore raw

Re: [PATCH] printk/console: Enable console kthreads only when there is no boot console left

Quoted 3 lines · On Tue, 21 Jun 2022 00:57:09 +0206 John Ogness <[email protected]>…

Since early printing does not use the kernel driver model, I do not see a clear solution at this point. But since using early printing is not a typical use, I am OK with this patch.

Does this patch still work if you add the "keep" flag to the earlyprintk kernel parameter?

-- Steve

maintainer
#17 lore raw

Re: [PATCH] printk/console: Enable console kthreads only when there is no boot console left

Quoted 9 lines · Steven Rostedt
Quoted 3 lines · On Tue, 21 Jun 2022 00:57:09 +0206 John Ogness <[email protected]>…

Since early printing does not use the kernel driver model, I do not see a clear solution at this point. But since using early printing is not a typical use, I am OK with this patch.

Does this patch still work if you add the "keep" flag to the earlyprintk kernel parameter?

It works. If the boot console stays then the console kthreads will never get enabled.

Well, I hope that we will be able to add the synchronization and remove this constraint. I think that the port->lock should be usable in most cases.

Best Regards,
Petr

#18 lore raw
In reply to Marek Behún

On (22/06/19 20:49), Marek Behún wrote:
[..]

Quoted 4 lines

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Can you please try disabling console kthreads and see how the boot process goes? Just return 0 from printk_activate_kthreads() (I think this should do the trick).

Quoted 20 lines

The stall example:

...

  [    1.108141] io scheduler mq-deadline registered
  [    1.108143] io scheduler kyber registered
  [    1.163484] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [

Or another:

...

  [    1.108155] io scheduler mq-deadline registered
  [    1.108156] io scheduler kyber registered
  [    1.162991] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [    1.163361] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
  [    1.180827] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
  [  

The garbage example:

Quoted 7 lines

...

  [    0.920951] raid6: using neon recovery algorithm
  [    0.921228] iommu: Default domain type: Translated
  %
          gb@k+cFL/[    4.954974] DSA: tree 0 setup
  [    4.955286] cfg80211: Loading compiled-in X.509 certificates for regulatory database

This is pretty suspicious. I don't see how console kthreads would corrupt the output. I suspect that something else is going on, some memory corruption, etc.

#19 lore raw
Quoted 10 lines · On Mon, 20 Jun 2022 14:23:28 +0900 Sergey Senozhatsky…

On (22/06/19 20:49), Marek Behún wrote:
[..]

Quoted 4 lines

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Can you please try disabling console kthreads and see how the boot process goes? Just return 0 from printk_activate_kthreads() (I think this should do the trick).

This indeed makes the problem go away...

Quoted 26 lines
Quoted 26 lines

The stall example:

...

  [    1.108141] io scheduler mq-deadline registered
  [    1.108143] io scheduler kyber registered
  [    1.163484] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [

Or another:

...

  [    1.108155] io scheduler mq-deadline registered
  [    1.108156] io scheduler kyber registered
  [    1.162991] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [    1.163361] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
  [    1.180827] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
  [  

The garbage example:

...

  [    0.920951] raid6: using neon recovery algorithm
  [    0.921228] iommu: Default domain type: Translated
  %
Quoted 6 lines
Quoted 2 lines
          gb@k+cFL/[    4.954974] DSA: tree 0 setup
  [    4.955286] cfg80211: Loading compiled-in X.509 certificates for regulatory database  

This is pretty suspicious. I don't see how console kthreads would corrupt the output. I suspect that something else is going on, some memory corruption, etc.

Maybe multiple threads are writing to serial registers, or something...

Marek

#20 lore raw
Quoted 7 lines · Marek Behún
Quoted 7 lines · Marek Behún

[..]

Quoted 4 lines

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Quoted 5 lines
Quoted 3 lines

Can you please try disabling console kthreads and see how the boot process goes? Just return 0 from printk_activate_kthreads() (I think this should do the trick).

This indeed makes the problem go away...

Oh... OK. Didn't expect that :)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

The garbage example:

...

  [    0.920951] raid6: using neon recovery algorithm
  [    0.921228] iommu: Default domain type: Translated
  %
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
          gb@k+cFL/[    4.954974] DSA: tree 0 setup
  [    4.955286] cfg80211: Loading compiled-in X.509 certificates for regulatory database  

This is pretty suspicious. I don't see how console kthreads would corrupt the output. I suspect that something else is going on, some memory corruption, etc.

Maybe multiple threads are writing to serial registers, or something...

That's possible. Console drivers usually should grab port->lock for write(), but maybe something is missing in the driver you use. What console driver are you using?

#21 lore raw
Quoted 30 lines · On Mon, 20 Jun 2022 19:13:37 +0900 Sergey Senozhatsky…
Quoted 7 lines · Marek Behún
Quoted 7 lines · Marek Behún

[..]

Quoted 4 lines

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Quoted 5 lines
Quoted 3 lines

Can you please try disabling console kthreads and see how the boot process goes? Just return 0 from printk_activate_kthreads() (I think this should do the trick).

This indeed makes the problem go away...

Oh... OK. Didn't expect that :)

Quoted 14 lines
Quoted 13 lines
Quoted 9 lines

The garbage example:

...

  [    0.920951] raid6: using neon recovery algorithm
  [    0.921228] iommu: Default domain type: Translated
  %
          gb@k+cFL/[    4.954974] DSA: tree 0 setup
  [    4.955286] cfg80211: Loading compiled-in X.509 certificates for regulatory database    

This is pretty suspicious. I don't see how console kthreads would corrupt the output. I suspect that something else is going on, some memory corruption, etc.

Quoted 5 lines
Quoted 1 line

Maybe multiple threads are writing to serial registers, or something...

That's possible. Console drivers usually should grab port->lock for write(), but maybe something is missing in the driver you use. What console driver are you using?

compatible = "snps,dw-apb-uart", so drivers/tty/serial/8250/8250_dw.c

seems that the function dw8250_serial_out() does not use the spinlock...

Marek

#22 lore raw

+Cc: Ilpo, a new maintainer for the driver

Quoted 45 lines · Marek Behún
Quoted 30 lines · On Mon, 20 Jun 2022 19:13:37 +0900 Sergey Senozhatsky…
Quoted 7 lines · Marek Behún
Quoted 7 lines · Marek Behún

[..]

Quoted 4 lines

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

Quoted 5 lines
Quoted 3 lines

Can you please try disabling console kthreads and see how the boot process goes? Just return 0 from printk_activate_kthreads() (I think this should do the trick).

This indeed makes the problem go away...

Oh... OK. Didn't expect that :)

Quoted 14 lines
Quoted 13 lines
Quoted 9 lines

The garbage example:

...

  [    0.920951] raid6: using neon recovery algorithm
  [    0.921228] iommu: Default domain type: Translated
  %
          gb@k+cFL/[    4.954974] DSA: tree 0 setup
  [    4.955286] cfg80211: Loading compiled-in X.509 certificates for regulatory database

This is pretty suspicious. I don't see how console kthreads would corrupt the output. I suspect that something else is going on, some memory corruption, etc.

Quoted 5 lines
Quoted 1 line

Maybe multiple threads are writing to serial registers, or something...

That's possible. Console drivers usually should grab port->lock for write(), but maybe something is missing in the driver you use. What console driver are you using?

compatible = "snps,dw-apb-uart", so drivers/tty/serial/8250/8250_dw.c

seems that the function dw8250_serial_out() does not use the spinlock...

Marek

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#23 lore raw
In reply to Marek Behún

Hi Marek,

Quoted 8 lines · Marek Behún

Hello Linus, Petr, John,

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

Quoted 7 lines

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

FWIW I have seen a similar issue to this on a totally different machine/board on arm (Sigmastar SSD202D, "mstarv7" in the kernel..) since rebasing my tree on 5.19.
I don't see lock ups all the time but every few boots the machine will deadlock. Usually with garbage as the last message on serial. With both serial and a display as consoles on the command line I could sometimes see more output on the display but I didn't manage to debug it.
Adding quiet to the command line made it go away so I've been doing that as workaround.

Serial driver in question is 8250_dw.c

Cheers,

Daniel

#24 lore raw
In reply to Marek Behún

[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]

Hi, this is your Linux kernel regression tracker.

Quoted 40 lines · Marek Behún

Hello Linus, Petr, John,

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

The stall example:

...

  [    1.108141] io scheduler mq-deadline registered
  [    1.108143] io scheduler kyber registered
  [    1.163484] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [

Or another:

...

  [    1.108155] io scheduler mq-deadline registered
  [    1.108156] io scheduler kyber registered
  [    1.162991] armada-ap806-pinctrl f06f4000.system-controller:pinctrl: registered pinctrl driver
  [    1.163361] armada-cp110-pinctrl f2440000.system-controller:pinctrl: registered pinctrl driver
  [    1.180827] mv_xor_v2 f0400000.xor: Marvell Version 2 XOR driver
  [  

The garbage example:

...

  [    0.920951] raid6: using neon recovery algorithm
  [    0.921228] iommu: Default domain type: Translated
  %
Quoted 6 lines
          gb@k+cFL/[    4.954974] DSA: tree 0 setup
  [    4.955286] cfg80211: Loading compiled-in X.509 certificates for regulatory database

I did not have time to look at the contents of the commit yet to try to fix it myself, and won't have time for another two weeks. But I thought I should let you know now.

CCing the regression mailing list, as it should be in the loop for all regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html

Thanks for the report. To be sure below issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced v5.18..v5.19-rc3
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply -- ideally with also telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/

Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (the mail this one replies to), as explained for in the Linux kernel's documentation; above webpage explains why this is important for tracked regressions.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.

#25 lore raw

Re: Boot stall regression from "printk for 5.19" merge #forregzbot

TWIMC: this mail is primarily send for documentation purposes and for regzbot, my Linux kernel regression tracking bot. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter.

Quoted 17 lines · Thorsten Leemhuis

[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]

Hi, this is your Linux kernel regression tracker.

Quoted 10 lines · Marek Behún

Hello Linus, Petr, John,

the series
https://lore.kernel.org/linux-kernel/YouKQw72H7y9EJQK@alley/
merged by commit 537e62c865dc ("Merge tag 'printk-for-5.19' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux")

and in particular the commit 09c5ba0aa2fc ("printk: add kthread console printers")

Quoted 6 lines
Quoted 4 lines

causes a regression on arm64 (Marvell CN9130-CRB board) where the system boot freezes in most cases (and is unusable until restarted by watchdog), or, in some cases boots, but the console output gets mangled for a while (the serial console spits garbage characters).

[...]
#regzbot ^introduced v5.18..v5.19-rc3

#regzbot fixed-by: 07a22b61946f0b80065b