New: Under heavy load FC TARGET going to Oops

https://bugzilla.kernel.org/show_bug.cgi?id=112241

Bug ID: 112241
Summary: Under heavy load FC TARGET going to Oops
Product: SCSI Drivers
Version: 2.5
Kernel Version: 4.3.3
Hardware: Intel
OS: Linux
Tree: Fedora
Status: NEW
Severity: high
Priority: P1
Component: QLOGIC QLA2XXX
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Created attachment 203261
--> https://bugzilla.kernel.org/attachment.cgi?id=203261&action=edit
Kernel stacktrace

Storage on Linux Fedora with QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA exporting Adaptec RAID6 with bcache on Intel SSD to VMWARE 5 On heavy load (for example VM migrating from storage) system going to Oops.

Stacktrace attached

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=112241

--- Comment #1 from Anthony <[email protected]> --- Created attachment 206351
--> https://bugzilla.kernel.org/attachment.cgi?id=206351&action=edit
Screenshot with call trace for kernel 4.5.0

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=112241

Anthony <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
Kernel Version|4.3.3 |4.5.0

--- Comment #2 from Anthony <[email protected]> --- With kernel 4.5.0 on target, system hang after clients connects to target.

You are receiving this mail because:
You are watching the assignee of the bug.

Hi Anthony,

So there are two things going on here.

First, the BUG_ON your ESX <-> LIO FC setup triggered has been addressed recently in v4.5-rc4 and later kernels with the following series:

http://www.spinics.net/lists/target-devel/msg11822.html

Note these patches will be making it back to earlier stable kernels over the next weeks.

However, this specific bug is a final consequence of larger ESX v5.5u2+ host side issue of AtomicTestandSet (ATS) heartbeat being enabled (by default) for all VMFS5 mounts:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113956

Other folks have been hitting this recently, here's some extra background:

http://permalink.gmane.org/gmane.linux.scsi.target.devel/11574
http://www.spinics.net/lists/target-devel/msg12124.html

Note this effects all targets w/ VAAI ATS (including EMC, IBM, 3PAR, SolidFire, etc) and the current solution for ESX v5.5u2+ is to either:

- Explicitly disable ATS heartbeat usage on all VMFS5 mounts as described in the VMWare -kb article, or:
- Explicitly disable all ATS logic completely from LIO using emulate_caw=0 on all backends connected to ESX v5.5u2+ hosts with VMFS5.

You can google for 'esx ats heartbeat bug' to see the gory details.

Thanks for reporting!

--nab

https://bugzilla.kernel.org/show_bug.cgi?id=112241

--- Comment #3 from nab <[email protected]> ---
Hi Anthony,

So there are two things going on here.

First, the BUG_ON your ESX <-> LIO FC setup triggered has been addressed recently in v4.5-rc4 and later kernels with the following series:

http://www.spinics.net/lists/target-devel/msg11822.html

Note these patches will be making it back to earlier stable kernels over the next weeks.

However, this specific bug is a final consequence of larger ESX v5.5u2+ host side issue of AtomicTestandSet (ATS) heartbeat being enabled (by default) for all VMFS5 mounts:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2113956

Other folks have been hitting this recently, here's some extra background:

http://permalink.gmane.org/gmane.linux.scsi.target.devel/11574
http://www.spinics.net/lists/target-devel/msg12124.html

Note this effects all targets w/ VAAI ATS (including EMC, IBM, 3PAR, SolidFire, etc) and the current solution for ESX v5.5u2+ is to either:

- Explicitly disable ATS heartbeat usage on all VMFS5 mounts as described in the VMWare -kb article, or:
- Explicitly disable all ATS logic completely from LIO using emulate_caw=0 on all backends connected to ESX v5.5u2+ hosts with VMFS5.

You can google for 'esx ats heartbeat bug' to see the gory details.

Thanks for reporting!

--nab

You are receiving this mail because:
You are watching the assignee of the bug.