Skip to contentSkip to conversation

New: arcconf host reset causes kernel panic -> driver crash?

Bug 207855 [email protected] 1 participant 15 messages May 22, 2020 – Nov 7, 2020 SCSI

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=207855

            Bug ID: 207855
           Summary: arcconf host reset causes kernel panic -> driver
                    crash?
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 5.6.13
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 289227
--> https://bugzilla.kernel.org/attachment.cgi?id=289227&action=edit
last dmesg captured

When performing a arcconf operation (assign hot-spare) on a adaptec 72405 SAS controller, the program crashes with the error "segmentation fault", but apparently, the driver is not too happy with it either: it becomes unresponsive, and makes it impossible to access scsi devices on the SAS controller.
Additional tricks to perform a PCI level reset ultimately lead to a kernel panic:
linuxserver# echo 1 > /sys/bus/pci/devices/0000\:04\:00.0/reset (wait a minute)
linuxserver# echo 1 > /sys/bus/pci/rescan
(wait a minute)
linuxserver# umount /data/* (where all SAS devices are mounted) (hangs indefinitely)
linuxserver# echo auto > /sys/bus/pci/devices/0000\:04\:00.0/power/control linuxserver# echo "0000:04:00.0" > /sys/bus/pci/drivers/aacraid/unbind --PANIC--

I haven't been able to C/P the panic output yet, working on a kexec kernel or crash dump.
The root directory is NOT one of the SAS devices, it is on a generic SATA controller

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#1 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

https://bugzilla.kernel.org/show_bug.cgi?id=207855

--- Comment #1 from Janpieter Sollie ([email protected]) ---
UPDATE: the host does not panic, but the whole IO system does not work any longer:
- network IO fails
- logon fails (hangs indefinitely)
- dmesg fails (hangs indefinitely)
- keyboard still works
I'd say a general IO error occurs (but why is there still USB keyboard input?), making the system unresponsive. Next time, I'll see whether I can still try a cat /dev/kmsg, but any use of kexec is off the table, I guess

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#3 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

Bart Van Assche ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from Bart Van Assche ([email protected]) --- Is this perhaps a recently introduced bug? If so, would it be possible to bisect this? See also
https://www.kernel.org/doc/html/latest/admin-guide/bug-bisect.html.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#4 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

--- Comment #4 from Janpieter Sollie ([email protected]) --- Good idea ... however, currently, I cleared + rebuilt the storage array, and everything is working again. Any idea what this segfault means so I can reproduce the state (host adapter reset) and cause the same error condition?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#5 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

--- Comment #5 from Bart Van Assche ([email protected]) ---
Is it possible to reproduce the kernel warning by running sg_reset -h /dev/sd... where /dev/sd... is a SCSI device controlled by an aacraid adapter? sg_reset is available in the sg3_utils package.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#6 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

--- Comment #6 from Janpieter Sollie ([email protected]) --- Sorry, I tried that, but IOP reset succeeded. I even tried it while the array was doing an expansion operation, but no luck. It came back with no issues

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#7 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

Janpieter Sollie ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |OBSOLETE

--- Comment #7 from Janpieter Sollie ([email protected]) --- I figured out it was due to an insufficient +5V line which made devices function the wrong way,I added some extra +5V juice and it worked without any problem.
Neverthless, is it an option to "isolate" the storage driver somewhat so the other PCI devices are kept up-and-running?
There are still some points of investigation:
-If it's PCI related, why does the dedicated VGA + onboard USB still work? -If it's storage subsystem related, why does network IO fail?
-If it's driver related, why is AHCI going down as well?
I guess this is not supposed to happen, so I'll see whether I can make it crash again, and eventually try to reset the whole PCI bus (and see whether that would help)

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#8 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

Janpieter Sollie ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|RESOLVED                    |REOPENED
         Resolution|OBSOLETE                    |---

--- Comment #8 from Janpieter Sollie ([email protected]) --- to update a bit:
I had the problem reoccured this morning. I can't access the PC right now, but I tried the remote syslog, and it displays something like:
... I know, the aacraid adapter panics, but why does it not reset the adapter and moves on? Why does the telnet daemon segfault in libc?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#10 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

--- Comment #10 from Janpieter Sollie ([email protected]) --- I just verified: the device was mostly dead:no F12 to enter kernel log, no num lock answer by kb led, but still replying to ping...
I currently locked the screen on tty12, so next time I *should* be able to see something

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#11 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

--- Comment #11 from Janpieter Sollie ([email protected]) --- the issue seems to be related to:

Quoted 10 lines
[59502.794967] Call Trace:
[59502.794967]  _raw_spin_lock_irqsave+0x20/0x30
[59502.794968]  __scsi_iterate_devices+0x22/0x80
[59502.794968]  scsi_eh_ready_devs+0x129/0x7c0
[59502.794968]  ? __pm_runtime_resume+0x54/0x70
[59502.794968]  scsi_error_handler+0x394/0x3a0
[59502.794969]  kthread+0xf3/0x130
[59502.794969]  ? scsi_eh_get_sense+0x120/0x120
[59502.794969]  ? kthread_park+0x80/0x80
[59502.794970]  ret_from_fork+0x1f/0x30

As far as I see, this stack blocks the entire scsi subsystem.
I do not see why: the scsi_error_handler runs in a separate kthread, so it *should* not block the IO subsystem ... but it definitely does: all storage devices on all SAS/SATA controllers (even USB) become inaccessible. I managed to get a dmesg out of it, but "echo 1 > /sys/class/pci_bus/0000\:04/device/reset"
never completed. this command was issued over a running SSH session. A new session could not be established any longer. But it proves the PCI subsystem is partially intact.

is it possible the raw_spin_lock_irqsave hurts when the adapter is not ready yet? and as such locks a device but never completes?

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#13 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

Janpieter Sollie ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

           See Also|                            |https://bugzilla.kernel.org
                   |                            |/show_bug.cgi?id=208605
Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#14 lore raw

[Bug 207855] arcconf host reset causes kernel panic -> driver crash?

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207855

Janpieter Sollie ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|REOPENED                    |RESOLVED
         Resolution|---                         |INVALID

--- Comment #12 from Janpieter Sollie ([email protected]) --- SAS adapter malfunction was due to a bad power supply - this was not a linux issue

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.