mvsas still has problems with 2.6.34

DISCUSSION Konstantinos Skarlatos 1 participant 1 message Jul 16, 2010

Subsystems (2)

SCSI MD RAID

lore

Share & tools

lore

Konstantinos Skarlatos

author

2010-07-16 09:34 UTC #0 lore raw

I am another user of mvsas, attached you can find two recent emails
with kernel logs that i have sent to the linux-scsi list regarding my problems with that driver

Kind regards

Quoted 253 lines · Thomas Fjellstrom

Quoted 8 lines · Caspar Smit
Quoted 8 lines · Caspar Smit
Quoted 7 lines
Thomas,
The patches you are using are the ones from november '09 i presume? Those
patches still had a lot of SATA issues so I think they didn't make the kernel. The patches seemed to handle SAS disks just fine though. SATA disks was a whole different story.
Quoted 5 lines
Quoted 3 lines
I'm actually using some that Andy Yan sent me privately, I'm not sure if they are the same exact ones he sent to linux-scsi. Probably are though.
The november patches were a set of 7 patches where only the first 6 needed to be applied.
Yeah, I was given a zip of the driver a little while before he posted the patches to the list.
Quoted 20 lines
Quoted 20 lines
Quoted 10 lines
Srinivas Naga Venkatasatya Pasagadugula created a patch instead of Andy Yan's patches which seemed to handle SATA disks a lot better but still after some tests it had alot of problems. Srinivas Naga Venkatasatya Pasagadugula is now in the process of creating a new patch to fix the remaining issues. He told me it would take a long time to create those and
it is now a few months ago since. I and others submitted extensive logging
for him to check.
Quoted 4 lines
As for production I could only advise this:
Using SAS disks: Use stock 2.6.34 kernel + Andy Yan's patches Using SATA disks: DO NOT GO INTO PRODCUTION.
I've been using the code Andy Yan sent me for 7 months now with 5 SATA disks
on a md raid5 array. I haven't noticed anything serious in that time. Prior
to tonight I had been using 2.6.32 for quite some time.
Quoted 6 lines
Quoted 2 lines
Maybe the issues only show up with serious load? My raid array doesn't get hammered, at least not often.
The main problem was hotplugging a SATA disk. This results in a kernel panic almost all of the time. There were more issues like the HDIO_GET_IDENTITY failed messages during boot for SATA disks and VERY SLOW xfs creation times.
I don't recall xfs taking /that/ long for a 4TB fs. With 2.6.34 I don't see any HDIO_GET_IDENTITY messages in dmesg. But I'll bet it freaks out if I try and hot remove one of the drives. I remember seeing the card lockup, and/or the kernel oopsing the last time I tried (2.6.30-2.6.32 time frame). Thankfully its not something I often do. While I Can, since I have a hot swap unit, its just not something I've had to do yet.
This array has been pretty solid for the past 6 months. Not sure it helps but I've been very careful with this machine, it gets shut down automatically and safely when the UPS battery gets low, so there hasn't been any abrupt shutdowns, except today when a forkbomb hit, and I had to SYSRQ+S+U+B the box.
At any rate I can help test whatever new patches might come along.
Quoted 169 lines
Kind regards,
Caspar Smit
Quoted 166 lines
Thanks
Quoted 164 lines
Kind regards,
Caspar Smit
Quoted 3 lines · Thomas Fjellstrom
Quoted 2 lines · Thomas Fjellstrom
Quoted 1 line
I've recently updated my server, and the mvsas driver included in
2.6.34.1 still causes my AOC-SASLP-MV8 card to completely lock up
Quoted 3 lines
Quoted 3 lines
after
Quoted 1 line
mdraid starts up on the devices. The machine is essentially in
"production" so I can't do a heck of a lot of testing on it anymore. The mvsas driver I got from Andy Yan seems to be a little outdated, it
Quoted 1 line
Quoted 1 line
Quoted 1 line
fails to compile due to a missing argument to
sas_change_queue_depth,
which I managed to fix, and I will try testing. I hope it works.
Quoted 3 lines
Quoted 1 line
It seems to work with the change I made.
Sorry for the noise, I forgot to post the following in my last couple
messages:
Quoted 3 lines
It works, but I do get a kernel warning:
Jul 16 00:38:05 boris kernel: [   20.104295] ------------[ cut here
]------------
Quoted 3 lines
Jul 16 00:38:05 boris kernel: [   20.104315] WARNING: at
drivers/ata/libata-core.c:5216 ata_qc_issue+0x31b/0x330 [libata]() Jul
16 00:38:05 boris kernel: [ 20.104323] Hardware name:
Quoted 3 lines
GA-MA790FXT-UD5P
Jul 16 00:38:05 boris kernel: [   20.104327] Modules linked in:
snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep
snd_pcm_oss
snd_mixer_oss nouveau ttm snd_pcm drm_kms_helper snd_seq_midi k10temp drm
Quoted 2 lines
agpgart i2c_algo_bit snd_rawmidi snd_seq_midi_event i2c_piix4 i2c_core
evdev edac_core edac_mce_amd tpm_tis snd_seq pcspkr tpm button tpm_bios wmi snd_timer snd_seq_device processor snd soundcore snd_page_alloc ext3 jbd mbcache dm_mod raid1 md_mod sg sr_mod sd_mod crc_t10dif cdrom ata_generic ohci_hcd ide_pci_generic ahci mvsas libsas libata atiixp scsi_transport_sas firewire_ohci firewire_core crc_itu_t thermal skge thermal_sys ide_core ehci_hcd r8169 mii usbcore scsi_mod nls_base [last unloaded: scsi_wait_scan]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104448] Pid: 6091, comm: ata_id
Not
tainted 2.6.34.1 #2
Quoted 2 lines
Jul 16 00:38:05 boris kernel: [   20.104453] Call Trace:
Jul 16 00:38:05 boris kernel: [   20.104462]  [<ffffffff81049bb3>] ?
warn_slowpath_common+0x73/0xb0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104472]  [<ffffffffa011686b>] ?
ata_qc_issue+0x31b/0x330 [libata]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104482]  [<ffffffffa000ef7f>] ?
scsi_init_io+0x2f/0x190 [scsi_mod]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104492]  [<ffffffffa011e020>] ?
ata_scsi_pass_thru+0x0/0x2e0 [libata]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104500]  [<ffffffffa0007990>] ?
scsi_done+0x0/0x20 [scsi_mod]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104509]  [<ffffffffa011bfae>] ?
ata_scsi_translate+0x9e/0x180 [libata]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104517]  [<ffffffffa0007990>] ?
scsi_done+0x0/0x20 [scsi_mod]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104525]  [<ffffffffa015522b>] ?
sas_queuecommand+0x9b/0x330 [libsas]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104533]  [<ffffffffa0007c7e>] ?
scsi_dispatch_cmd+0x17e/0x2b0 [scsi_mod]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104542]  [<ffffffffa000e830>] ?
scsi_request_fn+0x3e0/0x570 [scsi_mod]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104549]  [<ffffffff81058161>] ?
del_timer+0x71/0xd0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104556]  [<ffffffff811baed3>] ?
__blk_run_queue+0x63/0x130
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104563]  [<ffffffff811b43a2>] ?
elv_insert+0x132/0x1f0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104570]  [<ffffffff811bf1c9>] ?
blk_execute_rq_nowait+0x59/0xb0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104576]  [<ffffffff811bf292>] ?
blk_execute_rq+0x72/0xe0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104582]  [<ffffffff811bf05b>] ?
blk_rq_map_user+0x1ab/0x290
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104588]  [<ffffffff811c32f1>] ?
sg_io+0x241/0x3f0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104594]  [<ffffffff811c38fc>] ?
scsi_cmd_ioctl+0x45c/0x4b0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104601]  [<ffffffff8110e02f>] ?
__dentry_open+0x22f/0x340
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104607]  [<ffffffff811195b3>] ?
inode_permission+0x93/0xd0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.104614]  [<ffffffffa013cdc4>] ?
sd_ioctl+0xa4/0x120 [sd_mod]
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.105009]  [<ffffffff811c0798>] ?
__blkdev_driver_ioctl+0x98/0xe0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.105410]  [<ffffffff811c0c75>] ?
blkdev_ioctl+0x1f5/0x7b0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.105815]  [<ffffffff81113d30>] ?
cp_new_stat+0xe0/0x100
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.106230]  [<ffffffff8113b4f7>] ?
block_ioctl+0x37/0x40
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.106647]  [<ffffffff8111e985>] ?
vfs_ioctl+0x35/0xd0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.107064]  [<ffffffff8111ef08>] ?
do_vfs_ioctl+0x88/0x560
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.107490]  [<ffffffff8111402e>] ?
sys_newfstat+0x2e/0x50
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.107919]  [<ffffffff8111f460>] ?
sys_ioctl+0x80/0xa0
Quoted 1 line
Jul 16 00:38:05 boris kernel: [   20.108003]  [<ffffffff81002e2b>] ?
system_call_fastpath+0x16/0x1b
Quoted 6 lines
Jul 16 00:38:05 boris kernel: [   20.108003] ---[ end trace
e8ea9c22d6b28439 ]---
Other than this stack trace, it seems to work fine.
Quoted 1 line
Quoted 1 line
At some point though I really hope this gets fixed. I'm still
willing
to help test any new versions, just that I can't keep my box down for an extended period.
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Thanks.
I forgot to post, but here are the kernel messages I get when trying
to
use the kernel's included mvsas driver:
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.816129] sd 0:0:3:0: [sdf]
Unhandled
error code
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.816809] sd 0:0:3:0: [sdf]
Result:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.817470] sd 0:0:3:0: [sdf] CDB:
Read(10): 28 00 3a 45 c1 08 00 04 00 00
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.818853] sd 0:0:1:0: [sdd]
Unhandled
error code
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.819508] sd 0:0:1:0: [sdd]
Result:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.820179] sd 0:0:1:0: [sdd] CDB:
Read(10): 28 00 3a 45 be 58 00 02 b0 00
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.821558] sd 0:0:2:0: [sde]
Unhandled
error code
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.822201] sd 0:0:2:0: [sde]
Result:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.822836] sd 0:0:2:0: [sde] CDB:

Quoted 338 lines

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Read(10): 28 00 3a 45 c1 08 00 04 00 00
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.824157] sd 0:0:4:0: [sdg]
Unhandled
error code
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.824784] sd 0:0:4:0: [sdg]
Result:
Quoted 329 lines
Quoted 317 lines
Quoted 310 lines
hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Quoted 1 line
Jul 15 22:42:41 boris kernel: [  208.825407] sd 0:0:4:0: [sdg] CDB:
Read(10): 28 00 3a 45 c1 08 00 04 00 00
Quoted 10 lines
Jul 15 22:43:13 boris kernel: [  240.737334] md1_raid5     D
0000000000000001     0  6120      2 0x00000000
Jul 15 22:43:13 boris kernel: [  240.737948]  ffff88012c94c420
0000000000000046 ffff880100000000 ffff88012f65b680
Jul 15 22:43:13 boris kernel: [  240.738570]  00000000000134c0
ffff88012e6effd8 00000000000134c0 ffff88012c94c420
Jul 15 22:43:13 boris kernel: [  240.739196]  ffff88012e6effd8
ffff88012e6effd8 00000000000134c0 00000000000134c0
Jul 15 22:43:13 boris kernel: [  240.739821] Call Trace:
Jul 15 22:43:13 boris kernel: [  240.740458]  [<ffffffffa018d17e>] ?
md_super_wait+0xae/0xd0 [md_mod]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.741100]  [<ffffffff810671b0>] ?
autoremove_wake_function+0x0/0x30
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.741729]  [<ffffffffa018d748>] ?
md_update_sb+0x268/0x3d0 [md_mod]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.742361]  [<ffffffffa018fcd2>] ?
md_check_recovery+0x232/0x520 [md_mod]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.742982]  [<ffffffffa0421833>] ?
raid5d+0x23/0x4f0 [raid456]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.743602]  [<ffffffff8137883d>] ?
schedule_timeout+0x23d/0x310
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.744221]  [<ffffffff8103aee4>] ?
finish_task_switch+0x34/0xb0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.744861]  [<ffffffffa018ce43>] ?
md_thread+0x53/0x120 [md_mod]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.745489]  [<ffffffff810671b0>] ?
autoremove_wake_function+0x0/0x30
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.746121]  [<ffffffffa018cdf0>] ?
md_thread+0x0/0x120 [md_mod]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.746743]  [<ffffffff81066c9e>] ?
kthread+0x8e/0xa0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.747367]  [<ffffffff81003bd4>] ?
kernel_thread_helper+0x4/0x10
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.748000]  [<ffffffff81066c10>] ?
kthread+0x0/0xa0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.748639]  [<ffffffff81003bd0>] ?
kernel_thread_helper+0x0/0x10
Quoted 10 lines
Jul 15 22:43:13 boris kernel: [  240.750521] mount         D
0000000000000001     0  6405   6403 0x00000000
Jul 15 22:43:13 boris kernel: [  240.751158]  ffff88012eb8f3d0
0000000000000082 ffff88012e50c600 ffff88012f65d1c0
Jul 15 22:43:13 boris kernel: [  240.751805]  00000000000134c0
ffff88012dc0bfd8 00000000000134c0 ffff88012eb8f3d0
Jul 15 22:43:13 boris kernel: [  240.752452]  ffff88012dc0bfd8
ffff88012dc0bfd8 00000000000134c0 00000000000134c0
Jul 15 22:43:13 boris kernel: [  240.753108] Call Trace:
Jul 15 22:43:13 boris kernel: [  240.753761]  [<ffffffffa0020990>] ?
scsi_done+0x0/0x20 [scsi_mod]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.754409]  [<ffffffff8137883d>] ?
schedule_timeout+0x23d/0x310
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.755053]  [<ffffffff811ba097>] ?
blk_peek_request+0x127/0x1e0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.755708]  [<ffffffffa0020c8d>] ?
scsi_dispatch_cmd+0x18d/0x2b0 [scsi_mod]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.756358]  [<ffffffff81377af2>] ?
wait_for_common+0xd2/0x180
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.757023]  [<ffffffff8103da50>] ?
default_wake_function+0x0/0x20
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.757672]  [<ffffffffa041f486>] ?
unplug_slaves+0x86/0xc0 [raid456]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.758363]  [<ffffffffa048ed8d>] ?
xlog_bread_noalign+0xbd/0xf0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.759046]  [<ffffffffa04a38c0>] ?
xfs_buf_iowait+0x40/0xf0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.759730]  [<ffffffffa048ed8d>] ?
xlog_bread_noalign+0xbd/0xf0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.760423]  [<ffffffffa048edf5>] ?
xlog_bread+0x35/0x80 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.761124]  [<ffffffffa0491b9f>] ?
xlog_find_verify_cycle+0xbf/0x170 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.761813]  [<ffffffffa0492558>] ?
xlog_find_head+0x168/0x3a0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.762495]  [<ffffffffa04927b7>] ?
xlog_find_tail+0x27/0x3d0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.763178]  [<ffffffffa0492b75>] ?
xlog_recover+0x15/0x90 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.763858]  [<ffffffffa048b9c4>] ?
xfs_log_mount+0x134/0x170 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.764528]  [<ffffffffa0495b8f>] ?
xfs_mountfs+0x38f/0x720 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.765214]  [<ffffffffa04a090b>] ?
kmem_alloc+0x7b/0xc0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.765888]  [<ffffffffa04a09fb>] ?
kmem_zalloc+0x2b/0x40 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.766559]  [<ffffffffa04ad985>] ?
xfs_fs_fill_super+0x225/0x3b0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.767203]  [<ffffffff81112c03>] ?
get_sb_bdev+0x1a3/0x1e0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.767877]  [<ffffffffa04ad760>] ?
xfs_fs_fill_super+0x0/0x3b0 [xfs]
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.768533]  [<ffffffff81112633>] ?
vfs_kern_mount+0x83/0x1f0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.769174]  [<ffffffff81112813>] ?
do_kern_mount+0x53/0x120
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.769806]  [<ffffffff8112abfa>] ?
do_mount+0x28a/0x8a0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.770441]  [<ffffffff81128960>] ?
copy_mount_options+0xe0/0x180
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.771073]  [<ffffffff8112b2aa>] ?
sys_mount+0x9a/0xf0
Quoted 1 line
Jul 15 22:43:13 boris kernel: [  240.771695]  [<ffffffff81002e2b>] ?
system_call_fastpath+0x16/0x1b
Quoted 10 lines
Jul 15 22:45:13 boris kernel: [  360.769363] md1_raid5     D
0000000000000001     0  6120      2 0x00000000
Jul 15 22:45:13 boris kernel: [  360.770006]  ffff88012c94c420
0000000000000046 ffff880100000000 ffff88012f65b680
Jul 15 22:45:13 boris kernel: [  360.770648]  00000000000134c0
ffff88012e6effd8 00000000000134c0 ffff88012c94c420
Jul 15 22:45:13 boris kernel: [  360.771298]  ffff88012e6effd8
ffff88012e6effd8 00000000000134c0 00000000000134c0
Jul 15 22:45:13 boris kernel: [  360.771946] Call Trace:
Jul 15 22:45:13 boris kernel: [  360.772620]  [<ffffffffa018d17e>] ?
md_super_wait+0xae/0xd0 [md_mod]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.773265]  [<ffffffff810671b0>] ?
autoremove_wake_function+0x0/0x30
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.773911]  [<ffffffffa018d748>] ?
md_update_sb+0x268/0x3d0 [md_mod]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.774550]  [<ffffffffa018fcd2>] ?
md_check_recovery+0x232/0x520 [md_mod]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.775180]  [<ffffffffa0421833>] ?
raid5d+0x23/0x4f0 [raid456]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.775804]  [<ffffffff8137883d>] ?
schedule_timeout+0x23d/0x310
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.776424]  [<ffffffff8103aee4>] ?
finish_task_switch+0x34/0xb0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.777064]  [<ffffffffa018ce43>] ?
md_thread+0x53/0x120 [md_mod]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.777679]  [<ffffffff810671b0>] ?
autoremove_wake_function+0x0/0x30
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.778302]  [<ffffffffa018cdf0>] ?
md_thread+0x0/0x120 [md_mod]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.778919]  [<ffffffff81066c9e>] ?
kthread+0x8e/0xa0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.779534]  [<ffffffff81003bd4>] ?
kernel_thread_helper+0x4/0x10
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.780148]  [<ffffffff81066c10>] ?
kthread+0x0/0xa0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.780776]  [<ffffffff81003bd0>] ?
kernel_thread_helper+0x0/0x10
Quoted 10 lines
Jul 15 22:45:13 boris kernel: [  360.782623] mount         D
0000000000000001     0  6405   6403 0x00000000
Jul 15 22:45:13 boris kernel: [  360.783248]  ffff88012eb8f3d0
0000000000000082 ffff88012e50c600 ffff88012f65d1c0
Jul 15 22:45:13 boris kernel: [  360.783883]  00000000000134c0
ffff88012dc0bfd8 00000000000134c0 ffff88012eb8f3d0
Jul 15 22:45:13 boris kernel: [  360.784536]  ffff88012dc0bfd8
ffff88012dc0bfd8 00000000000134c0 00000000000134c0
Jul 15 22:45:13 boris kernel: [  360.785184] Call Trace:
Jul 15 22:45:13 boris kernel: [  360.785829]  [<ffffffffa0020990>] ?
scsi_done+0x0/0x20 [scsi_mod]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.786465]  [<ffffffff8137883d>] ?
schedule_timeout+0x23d/0x310
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.787098]  [<ffffffff811ba097>] ?
blk_peek_request+0x127/0x1e0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.787740]  [<ffffffffa0020c8d>] ?
scsi_dispatch_cmd+0x18d/0x2b0 [scsi_mod]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.788361]  [<ffffffff81377af2>] ?
wait_for_common+0xd2/0x180
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.788988]  [<ffffffff8103da50>] ?
default_wake_function+0x0/0x20
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.789612]  [<ffffffffa041f486>] ?
unplug_slaves+0x86/0xc0 [raid456]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.790277]  [<ffffffffa048ed8d>] ?
xlog_bread_noalign+0xbd/0xf0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.790933]  [<ffffffffa04a38c0>] ?
xfs_buf_iowait+0x40/0xf0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.791597]  [<ffffffffa048ed8d>] ?
xlog_bread_noalign+0xbd/0xf0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.792258]  [<ffffffffa048edf5>] ?
xlog_bread+0x35/0x80 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.792935]  [<ffffffffa0491b9f>] ?
xlog_find_verify_cycle+0xbf/0x170 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.793598]  [<ffffffffa0492558>] ?
xlog_find_head+0x168/0x3a0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.794258]  [<ffffffffa04927b7>] ?
xlog_find_tail+0x27/0x3d0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.794910]  [<ffffffffa0492b75>] ?
xlog_recover+0x15/0x90 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.795565]  [<ffffffffa048b9c4>] ?
xfs_log_mount+0x134/0x170 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.796216]  [<ffffffffa0495b8f>] ?
xfs_mountfs+0x38f/0x720 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.796879]  [<ffffffffa04a090b>] ?
kmem_alloc+0x7b/0xc0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.797527]  [<ffffffffa04a09fb>] ?
kmem_zalloc+0x2b/0x40 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.798171]  [<ffffffffa04ad985>] ?
xfs_fs_fill_super+0x225/0x3b0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.798785]  [<ffffffff81112c03>] ?
get_sb_bdev+0x1a3/0x1e0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.799429]  [<ffffffffa04ad760>] ?
xfs_fs_fill_super+0x0/0x3b0 [xfs]
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.800046]  [<ffffffff81112633>] ?
vfs_kern_mount+0x83/0x1f0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.800678]  [<ffffffff81112813>] ?
do_kern_mount+0x53/0x120
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.801292]  [<ffffffff8112abfa>] ?
do_mount+0x28a/0x8a0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.801910]  [<ffffffff81128960>] ?
copy_mount_options+0xe0/0x180
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.802531]  [<ffffffff8112b2aa>] ?
sys_mount+0x9a/0xf0
Quoted 1 line
Jul 15 22:45:13 boris kernel: [  360.803152]  [<ffffffff81002e2b>] ?
system_call_fastpath+0x16/0x1b
Quoted 2 lines
I'm pretty sure most of that is due to the driver not responding for 4
of
Quoted 10 lines
the drives (the first few messages)
Thanks again.
--
Thomas Fjellstrom
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi"
in
the body of a message to [email protected]
Quoted 7 lines
Quoted 1 line
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines
--
Thomas Fjellstrom
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hello all,

I have also got severe problems with mvsas, but have managed to at least make it usable. My config is a AOC-SASLP-MV8 with an HP SAS expander, with WD and Seagate sata disks connected to the expander using a norco 4020 case that does not have any on board expanders. Kernel version is 2.6.33 with latest srinivas patch. I also dont raid, the 13 disks have their own filesystem.

My experiences are:
The only filesystem that works is JFS. XFS and btrfs crash the controller making all the disks unreadable and that can only be solved by rebooting. (mkfs succeeds, crashes happen only after mounting or when fscking) JFS works ok as long as i access it via samba or make copies and moves with cp and mv. If I try file operations with Thunar file manager, the controller crashes. Smartctl has not caused any problems so far, and works ok.

Attached are some kernel logs captured from those crashes.

Kind regards,
Konstantinos Skarlatos

Quoted 36 lines · Jelle de Jong

Dear Srini,
I spent a few weeks catering information and did some intensive testing the last few days.
Srinivas Naga Venkatasatya Pasagadugula wrote, on 06-05-10 08:01:
Quoted 5 lines
1. Is this is the problem with only WD SATA drives? (I don't have WD SATA drives to reproduce this issue.)
2. Whether the problem is with direct attached SATA drives or drives connected in expanders also?
3. Could you please provide me the "dmesg" log or "/var/log/messages" log.
4. How much the capacity of SATA drives connected to controller?
5. Your HBA is having 6440 chipset?
With a few tricks I managed to boot my OS from the mvsas controller. I got eleven different sata disks attached from a 4-port mini-sas backplate without expander to the two Marvel 88SE63xx/64xx mvsas controllers in my system.
I managed to add five mdadm raid1 arrays without added the actual active sync device (so one disk for each array) I build my lvm systems on top of this and did a lot of file transfers for testing.
This worked stable enough, there are HDIO_GET_IDENTITY errors during boot and operation, but the hard disks seem to be working.
So to debug if issues are related to some brand of harddisk, I started to add WDC, Hitachi, SAMSUNG, Maxtor and Seagate disks to the respected raid1 array, the disks are of different sizes (320GB, 500GB, 1TB.)
The sync starts and will fail directly or a while later on failures in the mvsas driver. I attached an failure example as attachment.
The failures are of grave severity, I have lost complete lvm2 volumes and raid arrays during testing.
Do you also have a SuperMicro AOC-SASLP-MV8 controller for testing?

Quoted 7 lines

I would love to use the controllers for production, but they are currently unstable. I hope this information helps to solve the mvsas issues?
With kind regards,
Jelle de Jong

------------[ cut here ]------------
WARNING: at drivers/ata/libata-core.c:5186 ata_qc_issue+0x31f/0x330 [libata]()
Hardware name:
Modules linked in: ipv6 hwmon_vid jfs cpufreq_powersave fan cpufreq_ondemand edac_core powernow_k8 firewire_ohci psmouse firewire_core freq_table serio_raw pcspkr k8temp thermal crc_itu_t evdev edac_mce_amd skge processor button i2c_nforce2 sg forcedeth i2c_core fuse rtc_cmos rtc_core rtc_lib ext2 mbcache dm_crypt dm_mod ses enclosure sd_mod usb_storage ohci_hcd mvsas libsas sata_sil ehci_hcd scsi_transport_sas sata_nv usbcore pata_amd sata_via ata_generic pata_via pata_acpi libata scsi_mod
Pid: 3308, comm: smartctl Not tainted 2.6.33-ARCH #1
Call Trace:
 [<ffffffff810528c8>] warn_slowpath_common+0x78/0xb0
 [<ffffffff8105290f>] warn_slowpath_null+0xf/0x20
 [<ffffffffa002c14f>] ata_qc_issue+0x31f/0x330 [libata]
 [<ffffffffa0006fae>] ? scsi_init_sgtable+0x4e/0x90 [scsi_mod]
 [<ffffffffa0033cd0>] ? ata_scsi_pass_thru+0x0/0x2f0 [libata]
 [<ffffffffa00310c6>] ata_scsi_translate+0xa6/0x180 [libata]
 [<ffffffffa0000b10>] ? scsi_done+0x0/0x20 [scsi_mod]
 [<ffffffffa0000b10>] ? scsi_done+0x0/0x20 [scsi_mod]
 [<ffffffffa0034369>] ata_sas_queuecmd+0x139/0x2b0 [libata]
 [<ffffffffa00f3098>] sas_queuecommand+0x98/0x300 [libsas]
 [<ffffffffa0000c25>] scsi_dispatch_cmd+0xf5/0x230 [scsi_mod]
 [<ffffffffa0006ba2>] scsi_request_fn+0x322/0x3e0 [scsi_mod]
 [<ffffffff811b72bd>] __generic_unplug_device+0x2d/0x40
 [<ffffffff811bcbf8>] blk_execute_rq_nowait+0x68/0xb0
 [<ffffffff811bccc1>] blk_execute_rq+0x81/0xf0
 [<ffffffff811b4d0b>] ? blk_rq_bio_prep+0x2b/0xd0
 [<ffffffff811bc866>] ? blk_rq_map_kern+0xd6/0x150
 [<ffffffffa0007ee7>] scsi_execute+0xf7/0x160 [scsi_mod]
 [<ffffffffa0033167>] ata_cmd_ioctl+0x177/0x320 [libata]
 [<ffffffffa0033467>] ata_sas_scsi_ioctl+0x157/0x2b0 [libata]
 [<ffffffffa00f25f7>] sas_ioctl+0x47/0x50 [libsas]
 [<ffffffffa0002225>] scsi_ioctl+0xd5/0x390 [scsi_mod]
 [<ffffffffa0134d3e>] sd_ioctl+0xce/0xe0 [sd_mod]
 [<ffffffff811be35f>] __blkdev_driver_ioctl+0x8f/0xb0
 [<ffffffff811be82e>] blkdev_ioctl+0x22e/0x820
 [<ffffffff8114fdf7>] block_ioctl+0x37/0x40
 [<ffffffff81131ac8>] vfs_ioctl+0x38/0xd0
 [<ffffffff81131c70>] do_vfs_ioctl+0x80/0x560
 [<ffffffff811cff46>] ? __up_read+0xa6/0xd0
 [<ffffffff81077c29>] ? up_read+0x9/0x10
 [<ffffffff811321d1>] sys_ioctl+0x81/0xa0
 [<ffffffff8100a002>] system_call_fastpath+0x16/0x1b
---[ end trace 115ad6bf347654e7 ]---
 sdm: sdm1
 sdn: sdn1

INFO: task smbd:3348 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D ffff88000180f948 0 3348 2949 0x00000000
ffff88002dfe38b8 0000000000000086 0000000000000000 ffffffffa000692b 000000013d124bd0 0000000000011250 ffff88003caff938 ffff88003cf36690 000000010061522b ffff88002dfe3fd8 ffff88002dfe2000 ffff88002dfe2000

Call Trace:
 [<ffffffffa000692b>] ? scsi_request_fn+0xab/0x3e0 [scsi_mod]
 [<ffffffff810dbd00>] ? sync_page+0x0/0x50
 [<ffffffff8135970e>] io_schedule+0x6e/0xb0
 [<ffffffff810dbd3d>] sync_page+0x3d/0x50
 [<ffffffff81359d32>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dbce2>] __lock_page+0x62/0x70
 [<ffffffff81073090>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc3d9>] do_read_cache_page+0x159/0x180
 [<ffffffffa02e11e0>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dc434>] read_cache_page_async+0x14/0x20
 [<ffffffff810dc449>] read_cache_page+0x9/0x20
 [<ffffffffa02e1d85>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffffa02d4fb5>] diRead+0x155/0x200 [jfs]
 [<ffffffffa02c8d08>] jfs_iget+0x38/0x160 [jfs]
 [<ffffffffa02cb461>] jfs_lookup+0x71/0x140 [jfs]
 [<ffffffff81110000>] ? calculate_sizes+0x220/0x4a0
 [<ffffffff81359d53>] ? __wait_on_bit_lock+0x73/0xb0
 [<ffffffff8135a45d>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff8112bafb>] do_lookup+0x1db/0x270
 [<ffffffff8112e127>] link_path_walk+0x6b7/0xf10
 [<ffffffff810e1b28>] ? free_hot_page+0x28/0x90
 [<ffffffff8112eb1c>] path_walk+0x5c/0xc0
 [<ffffffff8112ecb3>] do_path_lookup+0x53/0xa0
 [<ffffffff8112f8f2>] user_path_at+0x52/0xa0
 [<ffffffff8115f01e>] ? locks_free_lock+0x3e/0x60
 [<ffffffff8115fb74>] ? fcntl_setlk+0x64/0x350
 [<ffffffff811260a7>] vfs_fstatat+0x37/0x70
 [<ffffffff81126206>] vfs_stat+0x16/0x20
 [<ffffffff8112622f>] sys_newstat+0x1f/0x50
 [<ffffffff81131370>] ? sys_fcntl+0x160/0x5d0
 [<ffffffff8100a002>] system_call_fastpath+0x16/0x1b

Here is one more log with crash information. All of my disks are
1.5TB, Seagates and WD. The expander did not affect the crashes, i had the same issues with direct attached drives.

Quoted 36 lines · Jelle de Jong

Dear Srini,
I spent a few weeks catering information and did some intensive testing the last few days.
Srinivas Naga Venkatasatya Pasagadugula wrote, on 06-05-10 08:01:
Quoted 5 lines
1. Is this is the problem with only WD SATA drives? (I don't have WD SATA drives to reproduce this issue.)
2. Whether the problem is with direct attached SATA drives or drives connected in expanders also?
3. Could you please provide me the "dmesg" log or "/var/log/messages" log.
4. How much the capacity of SATA drives connected to controller?
5. Your HBA is having 6440 chipset?
With a few tricks I managed to boot my OS from the mvsas controller. I got eleven different sata disks attached from a 4-port mini-sas backplate without expander to the two Marvel 88SE63xx/64xx mvsas controllers in my system.
I managed to add five mdadm raid1 arrays without added the actual active sync device (so one disk for each array) I build my lvm systems on top of this and did a lot of file transfers for testing.
This worked stable enough, there are HDIO_GET_IDENTITY errors during boot and operation, but the hard disks seem to be working.
So to debug if issues are related to some brand of harddisk, I started to add WDC, Hitachi, SAMSUNG, Maxtor and Seagate disks to the respected raid1 array, the disks are of different sizes (320GB, 500GB, 1TB.)
The sync starts and will fail directly or a while later on failures in the mvsas driver. I attached an failure example as attachment.
The failures are of grave severity, I have lost complete lvm2 volumes and raid arrays during testing.
Do you also have a SuperMicro AOC-SASLP-MV8 controller for testing?

Quoted 7 lines

I would love to use the controllers for production, but they are currently unstable. I hope this information helps to solve the mvsas issues?
With kind regards,
Jelle de Jong

drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1669:mvs_abort_task:rc= 5 drivers/scsi/mvsas/mv_sas.c 1608:mvs_query_task:rc= 5 sd 10:0:1:0: [sdc] Unhandled error code
sd 10:0:1:0: [sdc] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:1:0: [sdc] CDB: cdb[0]=0x28: 28 00 1c b2 86 6f 00 03 00 00
end_request: I/O error, dev sdc, sector 481461871
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 62 03 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442630912
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 ff 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442629888
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 fb 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442628864
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 f7 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442627840
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 f3 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442626816
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 1a 61 ef 00 00 04 00 00
end_request: I/O error, dev sdf, sector 442625792
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 00 00 c8 c8 00 00 08 00
end_request: I/O error, dev sdf, sector 51400
metapage_write_end_io: I/O error
sd 10:0:4:0: [sdf] Unhandled error code
sd 10:0:4:0: [sdf] Result: hostbyte=0x00 driverbyte=0x06 sd 10:0:4:0: [sdf] CDB: cdb[0]=0x2a: 2a 00 00 00 db 90 00 00 58 00
end_request: I/O error, dev sdf, sector 56208
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error
metapage_write_end_io: I/O error

INFO: task filezilla:18572 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
filezilla D ffffffff810dc2d0 0 18572 2063 0x00000000
ffff8800175b3768 0000000000000082 00000000000530d7 0000000000000008 0000000000800020 ffff88003ca52cb0 ffff8800175b3798 ffffffff811b6fa4 ffff8800175b3740 ffff8800175b3fd8 ffff8800175b2000 ffff8800175b2000

Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f87eb>] ? xtLookup+0x18b/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task filezilla:18574 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
filezilla D ffffffff810dc2d0 0 18574 2063 0x00000000
ffff88003c967768 0000000000000082 0000000000045ca7 0000000000000008 0000000000800020 ffff88003ca52cb0 ffff88003c967798 ffffffff811b6fa4 ffff88003c967740 ffff88003c967fd8 ffff88003c966000 ffff88003c966000

Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f86ed>] ? xtLookup+0x8d/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:18658 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D ffff88000190f948 0 18658 16782 0x00000000
ffff8800328cd608 0000000000000086 0000000000000000 ffffffff812772b2 ffff8800328cd5a8 ffffffffa000694a ffff880000000000 ffff88003c921830 00000001061958d0 ffff8800328cdfd8 ffff8800328cc000 ffff8800328cc000

Call Trace:
 [<ffffffff812772b2>] ? put_device+0x12/0x20
 [<ffffffffa000694a>] ? scsi_request_fn+0xaa/0x4d0 [scsi_mod]
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffffa0309851>] dtSearch+0x781/0xb00 [jfs]
 [<ffffffffa02f7503>] jfs_lookup+0x103/0x150 [jfs]
 [<ffffffff81137648>] ? d_alloc+0x158/0x1c0
 [<ffffffff8112c64d>] __lookup_hash+0xed/0x150
 [<ffffffff8112cb95>] lookup_one_len+0x75/0xb0
 [<ffffffffa03a114d>] vfsub_lookup_one_len+0x1d/0x50 [aufs]
 [<ffffffffa03a7ae2>] au_lkup_one+0x32/0xf0 [aufs]
 [<ffffffff810f9619>] ? handle_mm_fault+0x6d9/0xa30
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff8135d0c5>] ? __down_write_nested+0xd5/0xe0
 [<ffffffffa03a526f>] au_wh_test+0x1f/0xd0 [aufs]
 [<ffffffff8135bbc1>] ? mutex_lock+0x11/0x30
 [<ffffffffa03a8286>] au_lkup_dentry+0x336/0x530 [aufs]
 [<ffffffffa03af3cf>] aufs_lookup+0xef/0x230 [aufs]
 [<ffffffff8112c4cb>] do_lookup+0x1db/0x270
 [<ffffffff8112eb59>] link_path_walk+0x679/0xe50
 [<ffffffff8112f502>] path_walk+0x62/0xe0
 [<ffffffff8112f6c3>] do_path_lookup+0x53/0xa0
 [<ffffffff8112f735>] kern_path+0x25/0x50
 [<ffffffff811cb30b>] ? cpumask_any_but+0x2b/0x40
 [<ffffffff810fdbb0>] ? unmap_region+0x170/0x1a0
 [<ffffffff81151629>] lookup_bdev+0x39/0xb0
 [<ffffffff8112e26e>] ? getname+0x1ce/0x230
 [<ffffffff81174b03>] sys_quotactl+0xb3/0x3d0
 [<ffffffff81077df9>] ? up_write+0x9/0x10
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b

INFO: task filezilla:18572 blocked for more than 120 seconds.

Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f87eb>] ? xtLookup+0x18b/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task filezilla:18574 blocked for more than 120 seconds.

Call Trace:
 [<ffffffff811b6fa4>] ? generic_make_request+0x184/0x4f0
 [<ffffffff8107d739>] ? ktime_get_ts+0xa9/0xe0
 [<ffffffff810dc2d0>] ? sync_page+0x0/0x50
 [<ffffffff8135ad5e>] io_schedule+0x6e/0xb0
 [<ffffffff810dc30d>] sync_page+0x3d/0x50
 [<ffffffff8135b382>] __wait_on_bit_lock+0x52/0xb0
 [<ffffffff810dc2b2>] __lock_page+0x62/0x70
 [<ffffffff810732b0>] ? wake_bit_function+0x0/0x40
 [<ffffffff810dc9a9>] do_read_cache_page+0x159/0x180
 [<ffffffffa030d210>] ? metapage_readpage+0x0/0x180 [jfs]
 [<ffffffff810dca04>] read_cache_page_async+0x14/0x20
 [<ffffffff810dca19>] read_cache_page+0x9/0x20
 [<ffffffffa030ddb5>] __get_metapage+0x95/0x5a0 [jfs]
 [<ffffffff8135d1b4>] ? __down_read+0xd4/0xd6
 [<ffffffffa02f86ed>] ? xtLookup+0x8d/0x1a0 [jfs]
 [<ffffffffa0304227>] dbAlloc+0x147/0x480 [jfs]
 [<ffffffffa030cb92>] extAlloc+0x162/0x4d0 [jfs]
 [<ffffffff8111d740>] ? mem_cgroup_cache_charge+0x140/0x1e0
 [<ffffffffa02f4b21>] jfs_get_block+0x1c1/0x220 [jfs]
 [<ffffffff8114dd0a>] nobh_write_begin+0x1ea/0x4b0
 [<ffffffff8114ae16>] ? __set_page_dirty+0x76/0xd0
 [<ffffffffa02f461e>] jfs_write_begin+0x1e/0x20 [jfs]
 [<ffffffffa02f4960>] ? jfs_get_block+0x0/0x220 [jfs]
 [<ffffffff810db70d>] generic_file_buffered_write+0x10d/0x280
 [<ffffffff810581f2>] ? current_fs_time+0x22/0x30
 [<ffffffff810dd658>] __generic_file_aio_write+0x238/0x450
 [<ffffffff8135baad>] ? __mutex_lock_slowpath+0x26d/0x370
 [<ffffffff810dd8d4>] generic_file_aio_write+0x64/0xd0
 [<ffffffff81122002>] do_sync_write+0xd2/0x110
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8135dc8e>] ? common_interrupt+0xe/0x13
 [<ffffffff8119fcb1>] ? security_file_permission+0x11/0x20
 [<ffffffff81122b48>] vfs_write+0xb8/0x1a0
 [<ffffffff81122d0c>] sys_write+0x4c/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:18659 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D ffff88000180f948 0 18659 16782 0x00000000
ffff8800328e9b18 0000000000000086 0000000000000000 ffff8800328e9db8 0000000020000000 0000000000000000 0000000000000000 ffff88003c9209f0 000000010619a86e ffff8800328e9fd8 ffff8800328e8000 ffff8800328e8000

Call Trace:
 [<ffffffff8135d085>] __down_write_nested+0x95/0xe0
 [<ffffffff8135d0db>] __down_write+0xb/0x10
 [<ffffffff8135c6d9>] down_write+0x9/0x10
 [<ffffffffa03a74c7>] di_write_lock+0x27/0x50 [aufs]
 [<ffffffffa03983f6>] aufs_read_lock+0x106/0x110 [aufs]
 [<ffffffff81077e09>] ? up_read+0x9/0x10
 [<ffffffffa03afdd3>] ? aufs_permission+0x313/0x3d0 [aufs]
 [<ffffffffa03a8ae1>] aufs_d_revalidate+0x31/0x450 [aufs]
 [<ffffffff8112eab2>] link_path_walk+0x5d2/0xe50
 [<ffffffff8112f502>] path_walk+0x62/0xe0
 [<ffffffff8112f6c3>] do_path_lookup+0x53/0xa0
 [<ffffffff81130302>] user_path_at+0x52/0xa0
 [<ffffffff8113db14>] ? mntput_no_expire+0x24/0x100
 [<ffffffff811d1276>] ? __up_read+0xa6/0xd0
 [<ffffffff81077e09>] ? up_read+0x9/0x10
 [<ffffffff81126927>] vfs_fstatat+0x37/0x70
 [<ffffffff8113db14>] ? mntput_no_expire+0x24/0x100
 [<ffffffff81126a86>] vfs_stat+0x16/0x20
 [<ffffffff81126aaf>] sys_newstat+0x1f/0x50
 [<ffffffff8112c0ec>] ? path_put+0x2c/0x40
 [<ffffffff81120ee1>] ? sys_chdir+0x51/0x80
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b
INFO: task smbd:18660 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
smbd D ffff88000190f948 0 18660 16782 0x00000000
ffff88003a767d18 0000000000000086 0000000000000000 ffff88003a767d08 000001c73a767cb8 000001c8000001c8 000000083a767dd8 ffff88003c922d90 000000010619fccf ffff88003a767fd8 ffff88003a766000 ffff88003a766000

Call Trace:
 [<ffffffff8135d175>] __down_read+0x95/0xd6
 [<ffffffff8135c6e9>] down_read+0x9/0x10
 [<ffffffffa03a76b6>] di_read_lock+0x26/0x90 [aufs]
 [<ffffffffa03af5f2>] aufs_getattr+0xe2/0x4a0 [aufs]
 [<ffffffff8113030d>] ? user_path_at+0x5d/0xa0
 [<ffffffff811268bc>] vfs_getattr+0x4c/0x80
 [<ffffffff81126948>] vfs_fstatat+0x58/0x70
 [<ffffffff811269c9>] vfs_lstat+0x19/0x20
 [<ffffffff811269ef>] sys_newlstat+0x1f/0x50
 [<ffffffff81009fc2>] system_call_fastpath+0x16/0x1b