Skip to contentSkip to conversation

New: panic on write to 3ware raid array

Bug 90601 [email protected] 1 participant 30 messages Jan 2, 2015 – Oct 10, 2015 SCSI

lore
Share & tools
lore

[email protected]

author
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=90601

            Bug ID: 90601
           Summary: panic on write to 3ware raid array
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 3.16 and above
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 162291
--> https://bugzilla.kernel.org/attachment.cgi?id=162291&action=edit
panic log

Since kernel version 3.16 we are getting kernel panics and the panic message *might* suggest that it's related to the raid controller we use. I did try the kernel versions 3.17, 3.18 and also and it keeps crashing. 3.15.10 runs totally fine though.

I can reproduce the crash 100% with a simple DD command dd if=/dev/zero of=test bs=1G count=10

I've attached the backtrace and tw_cli configruation options. We have several cluster nodes so I can try patches or give additional information when necessary.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#3 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

kashyap <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #3 from kashyap <[email protected]> --- Merlin,

1.)
Can you enable CONFIG_DMA_API_DEBUG in your kernel conf and send complete /var/log/messages along with back trace.

2.)

Quoted 1 line

From existing logs I can figure out that crash happened at below place.

static void intel_unmap_sg(struct device *dev, struct scatterlist *sglist,
int nelems, enum dma_data_direction dir, struct dma_attrs *attrs)

{
        intel_unmap(dev, sglist[0].dma_address);
}

sglist[0] is NULL and while accessing dma_address (which is 0x10th index as per below definition)

struct scatterlist {

#ifdef CONFIG_DEBUG_SG
        unsigned long   sg_magic;
#endif
        unsigned long   page_link;
        unsigned int    offset;
        unsigned int    length;
        dma_addr_t      dma_address;   < -- 0x10th index
#ifdef CONFIG_NEED_SG_DMA_LENGTH
        unsigned int    dma_length;

#endif
};

Crash detail -

[ 1005.899543] Hardware name: Supermicro X9DR7/E-(J)LN4F/X9DR7/E-(J)LN4F, BIOS
3.0a 09/10/2013
[ 1005.899971] task: ffffffff8160a460 ti: ffffffff815f4000 task.ti:
ffffffff815f4000
[ 1005.900401] RIP: 0010:[<ffffffff813db181>]  [<ffffffff813db181>]

intel_unmap_sg+0x1/0x10

~ kashyap

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#4 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #4 from [email protected] ---
Hi Kashyap,

1. I enabled CONFIG_DMA_API_DEBUG , strangely enough after enabling it I couldn't even boot into the system anymore because it would crash when trying to access the root partition (after the kernel image was loaded). I think the problem was the initial xfs_repair that is done on bootup.

So I took the same kernel and copied it to a USB device and booted from that on the same host into a system that also resides on the USB disk.

Then I mounted /dev/sda (which is the first device on the raid controller which is the root partition).

Then I tried to reproduce the panic with the simple dd that was used before without success.
I then compiled the kernel sources with -j25 on the mounted root partition 2 times without any problems.
Then I chrooted onto that partition and did the same and could finally reproduce the crash.

I am not sure if that backtrace is more useful though.

I have also attached the /var/log/messages but of course that crash wasn't saved in it.
I also added my kernel config.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#9 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

Alex Elsayed <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |eternaleye+kernelbugs@gmail
                   |                            |.com

--- Comment #9 from Alex Elsayed <[email protected]> --- I've been bitten by this as well, on a 3ware 9750. It's currently preventing from using recent kernels on the server in question - which is an issue, because while 3.16.1 is rock-solid for scsi it has an intermittent wireless oops, and 3.17.1 oopses out with this (same backtrace as the existing attachment).

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#12 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #12 from Alex Elsayed <[email protected]> --- Created attachment 166681
--> https://bugzilla.kernel.org/attachment.cgi?id=166681&action=edit
Hardware of a second machine with the issue

Intel CPUs on an ASUS board - here's the output of lshw on the machine.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#13 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #13 from kashyap <[email protected]> --- My old email id is not working now, so I have to update my email id to new avagotech domain. Sorry for delay in response.

Does this mean issue does not happen with 3.16.1
but only happens with 3.17.1 (onwards) ? This is good data point.

Can you share whole dmesg logs for both 3.16.1 and 3.17.1 ?

Also are you using intel_iommu=on in your config ? Can you boot without enabling Intel IOMMU?

New logs with ""CONFIG_DMA_API_DEBUG" does not help much. You may have to post complete logs. We may have some interesting data before crash.

Thanks, Kashyap

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#14 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #14 from [email protected] --- @Kashyap

For me it started with 3.16 on all our servers with 3ware/lsi/avagotech controllers.
I rechecked in our internal ticket system.

We are using IOMMU so I completely disabled it, recompiled the kernal but getting the same error.

I also attached my kernel config before so maybe you can use that config and compile the kernel on your side.
Running with that config I can reproduce the crash with a simple dd command:
dd if=/dev/zero of=test bs=1G count=10

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#15 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #15 from kashyap <[email protected]> --- (In reply to merlin from comment #14)

Quoted 6 lines

@Kashyap

For me it started with 3.16 on all our servers with 3ware/lsi/avagotech controllers.
I rechecked in our internal ticket system.

Quoted 7 lines

We are using IOMMU so I completely disabled it, recompiled the kernal but getting the same error.

I also attached my kernel config before so maybe you can use that config and compile the kernel on your side.
Running with that config I can reproduce the crash with a simple dd command:
dd if=/dev/zero of=test bs=1G count=10

Can you share whole dmesg logs for the failure even if IOMMU is disabled ? I will try to reproduce at my setup as I was able to find same h/w.

~ Kashyap

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#17 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #17 from [email protected] ---
I am unable to get a dmesg output since the system crashes on boot. It looks like it's happening when udev is initialized.

Sorry for the screenshot but the system crashed before any network came up.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#18 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #18 from [email protected] ---
Created attachment 174171
--> https://bugzilla.kernel.org/attachment.cgi?id=174171&action=edit
dmesg

I finally managed to get a dmesg output with the crashing kernel.

Especially this part might indicate the root of the problem?

[Thu Apr 16 13:02:10 2015] ------------[ cut here ]------------
[Thu Apr 16 13:02:10 2015] WARNING: CPU: 6 PID: 1736 at lib/dma-debug.c:601
debug_dma_assert_idle+0x17c/0x1e0()
[Thu Apr 16 13:02:10 2015] 3w-sas 0000:03:00.0: DMA-API: cpu touching an active
dma mapped cacheline [cln=0x0000000101ff8e00]
[Thu Apr 16 13:02:10 2015] Modules linked in:
[Thu Apr 16 13:02:10 2015] CPU: 6 PID: 1736 Comm: mount Not tainted
4.0.0-gentoo #1
[Thu Apr 16 13:02:11 2015] Hardware name: Supermicro
X9DR7/E-(J)LN4F/X9DR7/E-(J)LN4F, BIOS 3.0a 09/10/2013
[Thu Apr 16 13:02:11 2015]  ffffffff8170e0f5 ffff88406050fcb8 ffffffff814c8778
000000000000009a
[Thu Apr 16 13:02:11 2015]  ffff88406050fd08 ffff88406050fcf8 ffffffff8107c250
ffff884060422e08
[Thu Apr 16 13:02:11 2015]  ffff8840647fe5e0 0000000000000206 ffffea00e1bf9ce8
ffff884060422e08
[Thu Apr 16 13:02:11 2015] Call Trace:
[Thu Apr 16 13:02:11 2015]  [<ffffffff814c8778>] dump_stack+0x45/0x57
[Thu Apr 16 13:02:11 2015]  [<ffffffff8107c250>] warn_slowpath_common+0x80/0xc0
[Thu Apr 16 13:02:11 2015]  [<ffffffff8107c2d1>] warn_slowpath_fmt+0x41/0x50
[Thu Apr 16 13:02:11 2015]  [<ffffffff812b9dfc>]
debug_dma_assert_idle+0x17c/0x1e0
[Thu Apr 16 13:02:11 2015]  [<ffffffff81114814>] do_wp_page+0xc4/0x810
[Thu Apr 16 13:02:11 2015]  [<ffffffff8111744c>] handle_mm_fault+0xb3c/0x10f0
[Thu Apr 16 13:02:11 2015]  [<ffffffff8111ca6d>] ? do_mmap_pgoff+0x34d/0x400
[Thu Apr 16 13:02:11 2015]  [<ffffffff81070ec6>] __do_page_fault+0x146/0x390
[Thu Apr 16 13:02:11 2015]  [<ffffffff8107114c>] do_page_fault+0xc/0x10
[Thu Apr 16 13:02:11 2015]  [<ffffffff814cf252>] page_fault+0x22/0x30
[Thu Apr 16 13:02:11 2015] ---[ end trace 326b0530baa65706 ]---

I was still able to login with root after that and reproduce the crash with another backtrace. See the next attachment.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#21 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

Justin Keogh <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #20 from Justin Keogh <[email protected]> --- Created attachment 174421
--> https://bugzilla.kernel.org/attachment.cgi?id=174421&action=edit
dmesg for the commit that git bisect landed on

I'm getting a similar panic (attached).

$lspci | grep 3ware
05:00.0 RAID bus controller: 3ware Inc 9750 SAS2/SATA-II RAID PCIe (rev 05)

$uname -a
Linux localhost 3.16.0-rc5+ #18 SMP PREEMPT Wed Apr 8 20:11:03 2015 x86_64 Intel(R) Xeon(R) CPU X5482 @ 3.20GHz GenuineIntel GNU/Linux

root@localhost /usr/src/linux $git rev-parse HEAD
74665016086615bbaa3fa6f83af410a0a4e029ee

The panic bisected to:
74665016086615bbaa3fa6f83af410a0a4e029ee is the first bad commit commit 74665016086615bbaa3fa6f83af410a0a4e029ee

Author: Christoph Hellwig <[email protected]>
Date:   Wed Jan 22 15:29:29 2014 +0100

scsi: convert host_busy to atomic_t

Avoid taking the host-wide host_lock to check the per-host queue limit. Instead we do an atomic_inc_return early on to grab our slot in the queue, and if necessary decrement it after finishing all checks.

I was unable to trigger the panic by writing to the 3w array with:
$dd if=/dev/zero of=test bs=1G count=10
or even:
$dd if=/dev/zero of=test bs=1G count=100

Instead, it takes somewhere between a few hours and 2 days of writing to panic, so the good bisects were run for 3 days to make sure.

Crash happens with and without ZFS modules loaded but most of the bisect was done with because I couldn't take the array out of the equation for a month. Both cases are attached along with .config and bisect log.

I'll recompile without IOMMU and post the results, further suggestions appreciated.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
author
#27 lore raw

[Bug 90601] panic on write to 3ware raid array

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=90601

--- Comment #25 from Justin Keogh <[email protected]> ---
Just to clarify, I couldn't trigger this by writing to the array from /dev/zero at all (maybe because ZFS), all panics were triggered by writing real data from another array on the same controller.

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.