Skip to contentSkip to conversation

New: Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

Bug 121531 [email protected] 3 participants 26 messages Jul 6, 2016 – Nov 1, 2016 SCSI

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=121531

            Bug ID: 121531
           Summary: Adaptec 7805H SAS HBA (pm80xx): hangs when writing
                    >80MB at once
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 3.16.0-4-amd64
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 222171
--> https://bugzilla.kernel.org/attachment.cgi?id=222171&action=edit
dd loop output, writing 64 - 128 MB to a disk

One of our customers attempted to install our Debian 8-based distribution on a Fujitsu PRIMERGY TX150 S8 server with an Adaptec 7805H SAS HBA. Unfortunately, the system tended to lock up during use; almost all services stopped responding, but it was still possible to run simple commands via SSH, e.g. "ssh server 'cat /proc/loadavg'" or "ssh server dmesg". Everything that required write access (like actually logging in via SSH, or using the web interface) just seemed to hang. Load average was extremely high (>100) and dmesg reported a lot of sas/pm80xx errors:

[11748.246360] sas: trying to find task 0xffff88082fcc7d40
[11748.246362] sas: sas_scsi_find_task: aborting task 0xffff88082fcc7d40
[11748.246572] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[11748.246574] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[11748.246576] sas: task done but aborted
[11748.246581] sas: sas_scsi_find_task: task 0xffff88082fcc7d40 is done
[11748.246583] sas: sas_eh_handle_sas_errors: task 0xffff88082fcc7d40 is done
[11748.246585] sas: trying to find task 0xffff88082fcc7c00
[11748.246587] sas: sas_scsi_find_task: aborting task 0xffff88082fcc7c00
[11748.246829] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[11748.246831] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[11748.246832] sas: task done but aborted
[11748.246837] sas: sas_scsi_find_task: task 0xffff88082fcc7c00 is done
[11748.246839] sas: sas_eh_handle_sas_errors: task 0xffff88082fcc7c00 is done
[11748.246841] sas: trying to find task 0xffff88082fcc7ac0
[11748.246844] sas: sas_scsi_find_task: aborting task 0xffff88082fcc7ac0
[11748.247055] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[11748.247057] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[11748.247059] sas: task done but aborted
[11748.247064] sas: sas_scsi_find_task: task 0xffff88082fcc7ac0 is done
[11748.247067] sas: sas_eh_handle_sas_errors: task 0xffff88082fcc7ac0 is done
[11748.247069] sas: trying to find task 0xffff88082fcc7840
[11748.247070] sas: sas_scsi_find_task: aborting task 0xffff88082fcc7840
[11748.247366] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[11748.247368] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[11748.247370] sas: task done but aborted
[11748.247375] sas: sas_scsi_find_task: task 0xffff88082fcc7840 is done
[11748.247377] sas: sas_eh_handle_sas_errors: task 0xffff88082fcc7840 is done
[11748.247379] sas: trying to find task 0xffff88082ff72e00
[11748.247380] sas: sas_scsi_find_task: aborting task 0xffff88082ff72e00
[11748.247591] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[11748.247593] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[11748.247595] sas: task done but aborted
[11748.247600] sas: sas_scsi_find_task: task 0xffff88082ff72e00 is done
[11748.247601] sas: sas_eh_handle_sas_errors: task 0xffff88082ff72e00 is done
[11748.247603] sas: trying to find task 0xffff88082ff72400
[11748.247605] sas: sas_scsi_find_task: aborting task 0xffff88082ff72400

At first we believed the underlying cause to be a hardware problem, but the problem persisted after the HBA and the backplane were replaced (the disks were ruled out as a possible cause because the selftests reported no errors).

To isolate the issue, I ran the following tests in a live system on the affected server:

1) "smartctl -t long" on both disks; both reported "Completed", so the disks seem to be OK.

2) "dd if=/dev/sdX of=/dev/null bs=1M" on both disks; both completed successfully, with an average speed of ~150 MB/s. Reading seems to be fine too.

3) "dd if=/dev/zero of=/dev/sdX bs=1M" on both disks. It stopped responding, and dmesg started spewing lots of sas/pm80xx errors. So apparently writing to the disks causes the problem.

To track it down further, I tried to repeatedly write 64 MB to one disk - this works without problems:

root@unassigned:~# for i in $(seq 1 8); do dd if=/dev/zero of=/dev/sdc bs=1M
count=64; done
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.482716 s, 139 MB/s
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.482339 s, 139 MB/s
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.474302 s, 141 MB/s
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.464919 s, 144 MB/s
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.465673 s, 144 MB/s
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.465525 s, 144 MB/s
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.473932 s, 142 MB/s
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.472965 s, 142 MB/s

Then I tried to write increasing amounts of data to the disk; this reproducibly slows down at about ~80 MB. A few seconds later, dmesg starts spewing error messages.

root@unassigned:~# for i in $(seq 64 128); do dd if=/dev/zero of=/dev/sdc bs=1M
count=$i; done
[...]
75+0 records in
75+0 records out
78643200 bytes (79 MB) copied, 0.595394 s, 132 MB/s
76+0 records in
76+0 records out
79691776 bytes (80 MB) copied, 33.6425 s, 2.4 MB/s
77+0 records in
77+0 records out
80740352 bytes (81 MB) copied, 0.631928 s, 128 MB/s
78+0 records in
78+0 records out
81788928 bytes (82 MB) copied, 0.621007 s, 132 MB/s
79+0 records in
79+0 records out
82837504 bytes (83 MB) copied, 0.651981 s, 127 MB/s
80+0 records in
80+0 records out
83886080 bytes (84 MB) copied, 0.674202 s, 124 MB/s
81+0 records in
81+0 records out
84934656 bytes (85 MB) copied, 33.7179 s, 2.5 MB/s
82+0 records in
82+0 records out
[...]

It seems to alternate between ~130 MB/sand 1-3 MB/s, and then completely hangs after 96 records. See dd-loop.txt for the full output. The errors in dmesg:

[ 2645.124944] sas: Enter sas_scsi_recover_host busy: 146 failed: 146
[ 2645.124963] sas: trying to find task 0xffff88083658b200
[ 2645.124966] sas: sas_scsi_find_task: aborting task 0xffff88083658b200
[ 2647.457375] sas: task done but aborted
[ 2647.457382] sas: task done but aborted
[ 2647.457385] sas: task done but aborted
[ 2647.457833] sas: task done but aborted
[ 2647.457840] sas: task done but aborted
[ 2647.457843] sas: task done but aborted
[ 2647.457851] sas: task done but aborted
[ 2647.457853] sas: task done but aborted
[ 2647.457856] sas: task done but aborted
[ 2647.457860] sas: task done but aborted
[ 2647.457863] sas: task done but aborted 
[ 2647.457865] sas: task done but aborted
[ 2647.457867] sas: task done but aborted
[ 2647.457869] sas: task done but aborted
[ 2647.457872] sas: task done but aborted
[ 2647.457874] sas: task done but aborted 
[ 2647.457876] sas: task done but aborted
[ 2647.457879] sas: task done but aborted 
[ 2647.457881] sas: task done but aborted
[ 2647.457883] sas: task done but aborted 
[ 2647.457885] sas: task done but aborted
[ 2647.458125] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[ 2647.458130] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[ 2647.458135] sas: task done but aborted 
[ 2647.458156] sas: sas_scsi_find_task: task 0xffff88083658b200 is done
[ 2647.458159] sas: sas_eh_handle_sas_errors: task 0xffff88083658b200 is done
[ 2647.458162] sas: trying to find task 0xffff880837ad30c0
[ 2647.458164] sas: sas_scsi_find_task: aborting task 0xffff880837ad30c0
[ 2647.458166] sas: sas_scsi_find_task: task 0xffff880837ad30c0 is done
[ 2647.458168] sas: sas_eh_handle_sas_errors: task 0xffff880837ad30c0 is done
[ 2647.458170] sas: trying to find task 0xffff880837ad3200
[ 2647.458172] sas: sas_scsi_find_task: aborting task 0xffff880837ad3200
[ 2647.458174] sas: sas_scsi_find_task: task 0xffff880837ad3200 is done
[ 2647.458176] sas: sas_eh_handle_sas_errors: task 0xffff880837ad3200 is done
[ 2647.458178] sas: trying to find task 0xffff880838dcfa80
[ 2647.458179] sas: sas_scsi_find_task: aborting task 0xffff880838dcfa80
[ 2647.458181] sas: sas_scsi_find_task: task 0xffff880838dcfa80 is done
[ 2647.458183] sas: sas_eh_handle_sas_errors: task 0xffff880838dcfa80 is done
[ 2647.458198] sas: trying to find task 0xffff880838d31700
[ 2647.458200] sas: sas_scsi_find_task: aborting task 0xffff880838d31700
[ 2647.458605] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[ 2647.458611] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[ 2647.458616] sas: task done but aborted
[ 2647.458638] sas: sas_scsi_find_task: task 0xffff880838d31700 is done
[ 2647.458641] sas: sas_eh_handle_sas_errors: task 0xffff880838d31700 is done
[ 2647.458644] sas: trying to find task 0xffff880838ca6e80
[ 2647.458646] sas: sas_scsi_find_task: aborting task 0xffff880838ca6e80
[ 2647.459184] pm80xx mpi_ssp_completion 1514:sas IO status 0x1
[ 2647.459190] pm80xx mpi_ssp_completion 1523:SAS Address of IO Failure
Drive:5000c50062c1b09d
[ 2647.459194] sas: task done but aborted
[ 2647.459217] sas: sas_scsi_find_task: task 0xffff880838ca6e80 is done
[ 2647.459220] sas: sas_eh_handle_sas_errors: task 0xffff880838ca6e80 is done
[ 2647.459222] sas: trying to find task 0xffff88083658b480
[ 2647.459225] sas: sas_scsi_find_task: aborting task 0xffff88083658b480

[...]

To finally rule out a hardware issue, I installed Windows 10 onto one of the disks and copied the Windows 10 installation image (~ 5 GB) from a USB stick onto the first disk, then formatted the second disk too and copied the image on that disk too. That worked without problems, so I'm pretty sure that this has to be a bug in the Linux driver.

I'll attach full dmesg copies, dmidecode/lspci/smartctl/uname output after filing this bug.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#6 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #6 from Martin von Wittich <[email protected]> --- Created attachment 222231
--> https://bugzilla.kernel.org/attachment.cgi?id=222231&action=edit
uname -a, cat /proc/version output

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#7 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #7 from Martin von Wittich <[email protected]> --- Created attachment 222241
--> https://bugzilla.kernel.org/attachment.cgi?id=222241&action=edit
dmesg output after booting, before writing to the disks

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#8 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #8 from Martin von Wittich <[email protected]> --- Created attachment 222251
--> https://bugzilla.kernel.org/attachment.cgi?id=222251&action=edit
dmesg output a few seconds after writing >128 MB to one disk

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#9 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #9 from Martin von Wittich <[email protected]> --- Created attachment 222261
--> https://bugzilla.kernel.org/attachment.cgi?id=222261&action=edit
dmesg output another few seconds later, completely filled with pm80xx errors

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#10 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #10 from Martin von Wittich <[email protected]> --- I forgot to mention: the issue is also reproducible on a Ubuntu 16.04 live system with Linux 4.4, by running "dd if=/dev/zero of=/dev/sdX bs=1M count=128".

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#11 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

Jack Wang <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #11 from Jack Wang <[email protected]> ---
Have you tried the version from microsemi? they have a lot of changes which are not
in upstream:
http://storage.microsemi.com/en-us/support/sas/sas/asa-7805h/

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#12 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #12 from Martin von Wittich <[email protected]> --- @Jack: No, I hadn't tried that yet, but I will now:

I downloaded the driver from the homepage and installed the adaptec/debian_7.4/x64/pm80xx-1.4.0-11068-debian64.deb package on our dev server. After DKMS had compiled the pm80xx.ko module, I unloaded the pm80xx module on the installer live system and replaced /lib/modules/3.16.0-4-amd64/kernel/drivers/scsi/pm8001/pm80xx.ko with the newly compiled module and loaded it again. It seems to work better... I've been able to write 1024 MB successfully:

root@(none):~# dd if=/dev/zero of=/dev/sdc bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 6.51408 s, 165 MB/s

I'm now installing Debian onto the machine, I'll report back how that turns out.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#13 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #13 from Martin von Wittich <[email protected]> --- Yup, the Debian installation worked and seems to work fine so far. I manually installed the pm80xx-1.4.0-11068-debian64.deb package in the target system while I was still in the installer by chrooting to /target; then I added "pm80xx" to /etc/initramfs/modules (without that update-initramfs wouldn't add the pm80xx module, I'm not really sure why).

After booting the system, I've succcessfully written ~500 GB of /dev/zero data into a file on an MD raid consisting of both of the disks. No error messages in the dmesg either.

Can you include those missing changes into the official kernel, or how can we resolve this bug? We'll ask to the customer if we can keep the server for an additional two weeks for testing, so if you need me to test builds, let me know.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#14 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #14 from Jack Wang <[email protected]> ---
The changes are huge, hard to do it, without help from MicroSemi/PMCs side. And I don't have hardware to test.

I've asked developer from MicroSemi to upsteam their changes, but sadly no reply.

2016-07-07 17:36 GMT+02:00 <[email protected]>:

Quoted 21 lines

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #13 from Martin von Wittich <[email protected]> --- Yup, the Debian installation worked and seems to work fine so far. I manually installed the pm80xx-1.4.0-11068-debian64.deb package in the target system while I was still in the installer by chrooting to /target; then I added "pm80xx" to /etc/initramfs/modules (without that update-initramfs wouldn't add the pm80xx module, I'm not really sure why).

After booting the system, I've succcessfully written ~500 GB of /dev/zero data into a file on an MD raid consisting of both of the disks. No error messages in the dmesg either.

Can you include those missing changes into the official kernel, or how can we resolve this bug? We'll ask to the customer if we can keep the server for an additional two weeks for testing, so if you need me to test builds, let me know.

--
You are receiving this mail because:
You are on the CC list for the bug.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#15 lore raw

Re: [Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

Patch set for pm80xx is pending for the last 3 quarters. We will submit those soon with all the buf fixes and performance tuning changes.

Regards,
Viswas G

Quoted 39 lines · On Thu, Jul 7, 2016 at 10:04 PM, <[email protected]> wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #14 from Jack Wang <[email protected]> ---
The changes are huge, hard to do it, without help from MicroSemi/PMCs side. And I don't have hardware to test.

I've asked developer from MicroSemi to upsteam their changes, but sadly no reply.

2016-07-07 17:36 GMT+02:00 <[email protected]>:

Quoted 13 lines

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #13 from Martin von Wittich <[email protected]> --- Yup, the Debian installation worked and seems to work fine so far. I manually installed the pm80xx-1.4.0-11068-debian64.deb package in the target system while I was still in the installer by chrooting to /target; then I added "pm80xx" to /etc/initramfs/modules (without that update-initramfs wouldn't add the pm80xx module, I'm not really sure why).

After booting the system, I've succcessfully written ~500 GB of /dev/zero data into a file on an MD raid consisting of both of the disks. No error messages in the dmesg either.

Quoted 8 lines

Can you include those missing changes into the official kernel, or how can we resolve this bug? We'll ask to the customer if we can keep the server for an additional two weeks for testing, so if you need me to test builds, let me know.

--
You are receiving this mail because:
You are on the CC list for the bug.

--
You are receiving this mail because:
You are the assignee for the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#16 lore raw

Re: [Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

Hi Viswas,

Thanks for update.
Good to know MicroSemi is still working on it!

Could you update the Maintainer file about your guys working email address?

Regards,
Jack

2016-07-08 6:47 GMT+02:00 Viswas G <[email protected]>:

Quoted 51 lines

Patch set for pm80xx is pending for the last 3 quarters. We will submit those soon with all the buf fixes and performance tuning changes.

Regards,
Viswas G

Quoted 32 lines · On Thu, Jul 7, 2016 at 10:04 PM, <[email protected]> wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #14 from Jack Wang <[email protected]> ---
The changes are huge, hard to do it, without help from MicroSemi/PMCs side. And I don't have hardware to test.

I've asked developer from MicroSemi to upsteam their changes, but sadly no reply.

2016-07-07 17:36 GMT+02:00 <[email protected]>:

Quoted 13 lines

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #13 from Martin von Wittich <[email protected]> --- Yup, the Debian installation worked and seems to work fine so far. I manually installed the pm80xx-1.4.0-11068-debian64.deb package in the target system while I was still in the installer by chrooting to /target; then I added "pm80xx" to /etc/initramfs/modules (without that update-initramfs wouldn't add the pm80xx module, I'm not really sure why).

After booting the system, I've succcessfully written ~500 GB of /dev/zero data into a file on an MD raid consisting of both of the disks. No error messages in the dmesg either.

Quoted 8 lines

Can you include those missing changes into the official kernel, or how can we resolve this bug? We'll ask to the customer if we can keep the server for an additional two weeks for testing, so if you need me to test builds, let me know.

--
You are receiving this mail because:
You are on the CC list for the bug.

Quoted 7 lines

--
You are receiving this mail because:
You are the assignee for the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#17 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #15 from Jack Wang <[email protected]> --- Hi Viswas,

Thanks for update.
Good to know MicroSemi is still working on it!

Could you update the Maintainer file about your guys working email address?

Regards,
Jack

2016-07-08 6:47 GMT+02:00 Viswas G <[email protected]>:

Quoted 51 lines

Patch set for pm80xx is pending for the last 3 quarters. We will submit those soon with all the buf fixes and performance tuning changes.

Regards,
Viswas G

Quoted 32 lines · On Thu, Jul 7, 2016 at 10:04 PM, <[email protected]> wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #14 from Jack Wang <[email protected]> ---
The changes are huge, hard to do it, without help from MicroSemi/PMCs side. And I don't have hardware to test.

I've asked developer from MicroSemi to upsteam their changes, but sadly no reply.

2016-07-07 17:36 GMT+02:00 <[email protected]>:

Quoted 13 lines

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #13 from Martin von Wittich <[email protected]> --- Yup, the Debian installation worked and seems to work fine so far. I manually installed the pm80xx-1.4.0-11068-debian64.deb package in the target system while I was still in the installer by chrooting to /target; then I added "pm80xx" to /etc/initramfs/modules (without that update-initramfs wouldn't add the pm80xx module, I'm not really sure why).

After booting the system, I've succcessfully written ~500 GB of /dev/zero data into a file on an MD raid consisting of both of the disks. No error messages in the dmesg either.

Quoted 8 lines

Can you include those missing changes into the official kernel, or how can we resolve this bug? We'll ask to the customer if we can keep the server for an additional two weeks for testing, so if you need me to test builds, let me know.

--
You are receiving this mail because:
You are on the CC list for the bug.

Quoted 7 lines

--
You are receiving this mail because:
You are the assignee for the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#18 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

Chloé Desoutter <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #16 from Chloé Desoutter <[email protected]> --- Hello,

I'm a victim of the bug and this impaired heavily the use of this controller on my filer when I wrote data constantly on it. After a few hours it would freeze without possible recovery.
I've been researching the cause of this bug by comparing the source trees of Microsemi and what's in the kernel.

The SCSI queue depth is not at stake, as changing it did not fix the issue. In the microsemi driver it is at 128, in the kernel tree it's at 508 but this changes nothing in the end (except I noticed a slight performance loss when set at 128).

However the MPI queue parameter is set way higher in the kernel tree than in the Microsemi tree.

In the Microsemi tree this is managed by the MAX_IB_QUEUE_ELEMENTS and MAX_OB_QUEUE_ELEMENTS defines. The events queue seems to be split evenly between reads and writes. The total queue length is 512. There is an equal number of inbound and outbound queues there.

In the kernel tree, this is handled by the PM8001_MPI_QUEUE define (value:
1024). There is 1 inbound queue and 4 outbound queues.

I noticed that the value PM8001_MPI_QUEUE = 1024 causes crashes of the driver on a "PMC-Sierra PM8001 SAS HBA" as reported earlier. Changing this value to 512 results in a much more stable driver. I guess setting the MPI queue to something too important results in instructions being lost when too much data gets queued and the controller cannot keep up with the writes.

I will attach the following patch.

--- linux/drivers/scsi/pm8001/pm8001_defs.h.orig 2016-10-25
15:15:40.470112331 +0000
+++ linux/drivers/scsi/pm8001/pm8001_defs.h 2016-10-24 19:13:46.533108727
+0000
@@ -76,7 +76,7 @@ enum port_type {

/* driver compile-time configuration */
#define PM8001_MAX_CCB 512 /* max ccbs supported */
-#define PM8001_MPI_QUEUE 1024 /* maximum mpi queue entries */
+#define PM8001_MPI_QUEUE 512 /* maximum mpi queue entries */

 #define    PM8001_MAX_INB_NUM     1
 #define    PM8001_MAX_OUTB_NUM     1
 #define    PM8001_MAX_SPCV_INB_NUM        1
Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#19 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #17 from Chloé Desoutter <[email protected]> --- Created attachment 242701
--> https://bugzilla.kernel.org/attachment.cgi?id=242701&action=edit
pm8001_defs.h patch

PM8001_MPI_QUEUE: 1024 → 512 (more stable)

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#20 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #18 from Chloé Desoutter <[email protected]> --- Actually further MicroSemi driver analysis lets me think that the proper, recommended value is 256.

$ egrep '#define\s+MAX_[IO]B_QUEUE_ELEMENTS' *.h
pm8001_sas.h:#define    MAX_IB_QUEUE_ELEMENTS   256
pm8001_sas.h:#define    MAX_OB_QUEUE_ELEMENTS   256

so I shall test with this value for PM8001_MPI_QUEUE and see if I achieve real stability with constant workloads.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#21 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #19 from Chloé Desoutter <[email protected]> --- Said parameter has been introduced by this commit. 99c72ebceb4dda445b4b74c6f46035feec95a2b3

The rationale is OK but the flaw is that sometimes the controller will crash completely so there needs to be another way.

I suggest we set the PM8001_MPI_QUEUE to 256 and find another way to mitigate these performance degradations because we cannot afford crashed SAS controllers.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#23 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

Chloé Desoutter <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

 Attachment #242701|0                           |1

is obsolete| |

--- Comment #21 from Chloé Desoutter <[email protected]> --- Created attachment 242801
--> https://bugzilla.kernel.org/attachment.cgi?id=242801&action=edit
pm8001_defs.h patch [2]

Set PM8001_MPI_QUEUE to 256, as in the MicroSemi driver.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#24 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #22 from Chloé Desoutter <[email protected]> --- (In reply to Chloé Desoutter from comment #20)

Quoted 3 lines

Currently testing w/ PM8001_MPI_QUEUE = 256.

Prospective patch attached.

I confirm that after 36 hours of intensive workload, I see no visible performance loss on a PM8001 and that there's been no data error since then.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#25 lore raw

[Bug 121531] Adaptec 7805H SAS HBA (pm80xx): hangs when writing >80MB at once

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=121531

--- Comment #23 from Chloé Desoutter <[email protected]> --- I can trigger crashes after a long while in a heavy workload, quite randomly, with 256.

Out of curiosity I checked the pmspcv driver from FreeBSD and they use a lower value still :

#define MPI_MAX_INBOUND_QUEUES 64 /**< Maximum number of inbound
queues */
#define MPI_MAX_OUTBOUND_QUEUES 64 /**< Maximum number of outbound
queues */

                                               /**< Max # of memory chunks

supported */
#define MPI_MAX_MEM_REGIONS (MPI_MAX_INBOUND_QUEUES +
MPI_MAX_OUTBOUND_QUEUES) + 4
#define MPI_LOGSIZE 4096 /**< default size */

so I'll try with this value and give feedback for stability and performance.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.