WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
Hi,
During recent debugging session of my SCSI target SCST (http://scst.sf.net) I noticed many
WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
messages in kernel log on the initiator. I attached the full log of several of them.
My target was buggy and I was working on fixing it, but I suppose Linux should handle such failures more gracefully. In all the cases the target had one type of failure: it "ate" a SCSI command and never returned result of it.
Low level drivers were open-iscsi and qla2xxx.
Vlad
Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 816 Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_get_inode_loc: unable to read inode block - inode=2, block=316 Oct 16 21:58:54 inif kernel: ------------[ cut here ]------------ Oct 16 21:58:54 inif kernel: WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66() Oct 16 21:58:54 inif kernel: Modules linked in: crc32c crypto_hash libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi firmware_class scsi_transport_fc tcp_highspeed w83627hf hwmon_vid eeprom adm1021 hwmon binfmt_misc dm_mirror dm_log dm_mod pci_slot battery ac ehci_hcd uhci_hcd sg e1000 pcspkr i2c_i801 i2c_core ide_cd_mod cdrom e7xxx_edac button usbcore [last unloaded: qla2xxx] Oct 16 21:58:54 inif kernel: Pid: 19064, comm: pam_console_app Not tainted 2.6.27-ini #3 Oct 16 21:58:54 inif kernel: [<b0116065>] warn_on_slowpath+0x4d/0x66 Oct 16 21:58:54 inif kernel: [<b01270bd>] enqueue_hrtimer+0x72/0xea Oct 16 21:58:54 inif kernel: [<b0112adb>] hrtick_start_fair+0x8e/0xd9 Oct 16 21:58:54 inif kernel: [<b0112dc3>] pick_next_task_fair+0x8c/0xa2 Oct 16 21:58:54 inif kernel: [<b03098fa>] schedule+0x173/0x3e9 Oct 16 21:58:54 inif kernel: [<b017a205>] mark_buffer_dirty+0x51/0x66 Oct 16 21:58:54 inif kernel: [<b01a5abc>] ext3_commit_super+0x3a/0x4e Oct 16 21:58:54 inif kernel: [<b01a6603>] ext3_handle_error+0x62/0xaa Oct 16 21:58:54 inif kernel: [<b0116d25>] printk+0x1b/0x1f Oct 16 21:58:54 inif kernel: [<b01a66fd>] ext3_error+0x55/0x5a Oct 16 21:58:54 inif kernel: [<b019e90f>] __ext3_get_inode_loc+0x253/0x2b9 Oct 16 21:58:54 inif kernel: [<b019e99f>] ext3_reserve_inode_write+0x19/0x5f Oct 16 21:58:54 inif kernel: [<b0167e7c>] filldir64+0x0/0xc5 Oct 16 21:58:54 inif kernel: [<b019e9ff>] ext3_mark_inode_dirty+0x1a/0x30 Oct 16 21:58:54 inif kernel: [<b01ae8ec>] journal_start+0x80/0xa9 Oct 16 21:58:54 inif kernel: [<b01a16d6>] ext3_dirty_inode+0x65/0x6e Oct 16 21:58:54 inif kernel: [<b0176b31>] __mark_inode_dirty+0x24/0x133 Oct 16 21:58:54 inif kernel: [<b0167e7c>] filldir64+0x0/0xc5 Oct 16 21:58:54 inif kernel: [<b016eb53>] touch_atime+0xa1/0xee Oct 16 21:58:54 inif kernel: [<b0167e7c>] filldir64+0x0/0xc5 Oct 16 21:58:54 inif kernel: [<b0168077>] vfs_readdir+0x7c/0x7e Oct 16 21:58:54 inif kernel: [<b01680dc>] sys_getdents64+0x63/0xa5 Oct 16 21:58:54 inif kernel: [<b0102cb9>] sysenter_do_call+0x12/0x21 Oct 16 21:58:54 inif kernel: ======================= Oct 16 21:58:54 inif kernel: ---[ end trace ddb1531dc775930c ]--- Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf) in ext3_reserve_inode_write: IO failure Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:58:54 inif kernel: lost page write due to I/O error on sdf Oct 16 21:58:54 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:58:54 inif last message repeated 157 times Oct 16 21:59:05 inif kernel: journal_bmap: journal block not found at offset 859 on sdf Oct 16 21:59:05 inif kernel: Aborting journal on device sdf. Oct 16 21:59:05 inif kernel: __ratelimit: 158 callbacks suppressed Oct 16 21:59:05 inif kernel: Buffer I/O error on device sdf, logical block 822 Oct 16 21:59:05 inif kernel: lost page write due to I/O error on sdf Oct 16 21:59:05 inif kernel: journal commit I/O error Oct 16 21:59:05 inif kernel: sd 12:0:0:0: [sdb] Synchronizing SCSI cache Oct 16 21:59:05 inif kernel: sd 12:0:0:2: [sdd] Synchronizing SCSI cache Oct 16 21:59:05 inif kernel: sd 12:0:0:3: [sde] Synchronizing SCSI cache Oct 16 21:59:12 inif kernel: EXT3-fs error (device sdf): ext3_readdir: directory #2 contains a hole at offset 0 Oct 16 21:59:12 inif kernel: Buffer I/O error on device sdf, logical block 0 Oct 16 21:59:12 inif kernel: lost page write due to I/O error on sdf Oct 16 21:59:12 inif kernel: ext3_abort called. Oct 16 21:59:12 inif kernel: EXT3-fs error (device sdf): ext3_journal_start_sb: Detected aborted journal Oct 16 21:59:12 inif kernel: Remounting filesystem read-only Oct 16 21:59:44 inif kernel: Buffer I/O error on device sdf, logical block 822 Oct 16 21:59:44 inif kernel: lost page write due to I/O error on sdf
Quoted 7 lines · Vladislav Bolkhovitin
Hi,
During recent debugging session of my SCSI target SCST
(http://scst.sf.net) I noticed manyWARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()
Quoted 7 lines
messages in kernel log on the initiator. I attached the full log of several of them.
My target was buggy and I was working on fixing it, but I suppose Linux should handle such failures more gracefully. In all the cases the target had one type of failure: it "ate" a SCSI command and never returned result of it.
Right. This is one of the warnings I see in my fault-injection testing. It is fixed by my patch to clean up and improve the page and buffer error handling in the vm/fs.
Quoted 11 lines · Nick Piggin
Quoted 10 lines · Vladislav Bolkhovitin
Hi,
During recent debugging session of my SCSI target SCST
(http://scst.sf.net) I noticed manyWARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()messages in kernel log on the initiator. I attached the full log of several of them.
Quoted 8 lines
Quoted 4 lines
My target was buggy and I was working on fixing it, but I suppose Linux should handle such failures more gracefully. In all the cases the target had one type of failure: it "ate" a SCSI command and never returned result of it.
Right. This is one of the warnings I see in my fault-injection testing. It is fixed by my patch to clean up and improve the page and buffer error handling in the vm/fs.
Can you specify which patch you referring? Is it in 2.6.27?
Quoted 17 lines · Vladislav Bolkhovitin
Quoted 16 lines · Nick Piggin
Quoted 14 lines · Vladislav Bolkhovitin
Hi,
During recent debugging session of my SCSI target SCST
(http://scst.sf.net) I noticed manyWARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()messages in kernel log on the initiator. I attached the full log of several of them.
My target was buggy and I was working on fixing it, but I suppose Linux should handle such failures more gracefully. In all the cases the target had one type of failure: it "ate" a SCSI command and never returned result of it.
Quoted 5 lines
Quoted 3 lines
Right. This is one of the warnings I see in my fault-injection testing. It is fixed by my patch to clean up and improve the page and buffer error handling in the vm/fs.
Can you specify which patch you referring? Is it in 2.6.27?
It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
Quoted 17 lines · Nick Piggin
Quoted 16 lines · Vladislav Bolkhovitin
Quoted 15 lines · Nick Piggin
Quoted 14 lines · Vladislav Bolkhovitin
Hi,
During recent debugging session of my SCSI target SCST
(http://scst.sf.net) I noticed manyWARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()messages in kernel log on the initiator. I attached the full log of several of them.
My target was buggy and I was working on fixing it, but I suppose Linux should handle such failures more gracefully. In all the cases the target had one type of failure: it "ate" a SCSI command and never returned result of it.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Right. This is one of the warnings I see in my fault-injection testing. It is fixed by my patch to clean up and improve the page and buffer error handling in the vm/fs.
Can you specify which patch you referring? Is it in 2.6.27?
It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
I see. I'm looking forward to see it in 2.6.28 or .29. This is really a needed work.
BTW, have you even seen in your fault-injection testing that after receiving a failure from a SCSI device during heavy load ext3 file system mounted on it gets corrupted and journal replay on remount doesn't repair it, only manual e2fsck helps? I've many times seen that, including cases when the target was remaining up and fully functional. See, e.g., "MOANING MODE ON" part in http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked that case since then, although I see such corruptions quite often. But in all them I can't so clearly say that it isn't a target's failure.
Vlad
Re: FS corruption after I/O errors
Quoted 21 lines · Vladislav Bolkhovitin
Quoted 20 lines · Nick Piggin
Quoted 19 lines · Vladislav Bolkhovitin
Quoted 18 lines · Nick Piggin
Quoted 7 lines · Vladislav Bolkhovitin
Hi,
During recent debugging session of my SCSI target SCST
(http://scst.sf.net) I noticed manyWARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()Quoted 7 lines
messages in kernel log on the initiator. I attached the full log of several of them.
My target was buggy and I was working on fixing it, but I suppose Linux should handle such failures more gracefully. In all the cases the target had one type of failure: it "ate" a SCSI command and never returned result of it.
Right. This is one of the warnings I see in my fault-injection testing. It is fixed by my patch to clean up and improve the page and buffer error handling in the vm/fs.
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
Can you specify which patch you referring? Is it in 2.6.27?
It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
I see. I'm looking forward to see it in 2.6.28 or .29. This is really a needed work.
BTW, have you even seen in your fault-injection testing that after receiving a failure from a SCSI device during heavy load ext3 file system mounted on it gets corrupted and journal replay on remount doesn't repair it, only manual e2fsck helps? I've many times seen that, including cases when the target was remaining up and fully functional. See, e.g., "MOANING MODE ON" part in http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked that case since then, although I see such corruptions quite often. But in all them I can't so clearly say that it isn't a target's failure.
I've just checked it with 2.6.27. The situation greatly improved and dbench was able to complete several runs under constant TASK_ABORTED "bombarding" (TASK RESET task management commands using "sg_reset -b" each 31 seconds from another "connection" to that device via qla2xxx initiator driver. You can see those resets in the attached log). But when then I unmounted the affected partition, e2fsck found errors on it. See attachments for details. The target all the times was fine and completely healthy.
[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information/dev/sdc: 69413/640000 files (0.2% non-contiguous), 316508/1280000 blocks
[root@inif w]# mount -t ext3 /dev/sdc /mnt
[root@inif dbench-mod]# while true; do ./dbench 50; done
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...................................+..........................................................................+.................................................................+.+.................................................+.....................................+................................................................................................................++...........................++....................++..........................+.......................+......................................+.................................................................................................+.........................++++.+...+................................+...++.+........++..........+.................++++......................+++....+.++.....+.....+.....................+......+++.++++**************************************************
Throughput 27.5109 MB/sec (NB=34.3887 MB/sec 275.109 MBit/sec)
Average latency 7.166755 ms
Maximum latency 22972.843000 ms
Latency > 2 sec. 0.091621 %
Latency > 10 sec. 0.016009 %
50 clients started
............................................................................................................write failed on handle 4290 ..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+............................................................................................................................................................................................................................................................................................................................................................................................................................++.........+......................................write failed on handle 4279
........................+.................................................+...................................................+...............+.........+...............................+.............++..............................++...........................+++........................+...+............+......+...+.+......+.....+...............++.....+....................+.....+..+++.+............+....+++......+.+...++.....+........+........+.+++++**************************************************
Throughput 27.5921 MB/sec (NB=34.4902 MB/sec 275.921 MBit/sec)
Average latency 7.097532 ms
Maximum latency 20892.153000 ms
Latency > 2 sec. 0.087833 %
Latency > 10 sec. 0.016437 %
50 clients started
....................................................................................write failed on handle 4290
..(4727) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4729) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4328 (File exists)
(4730) nb_close: handle 4328 was not open
(4739) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4743) nb_stat: CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP size=260096 No such file or directory
(4745) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4329 (File exists)
(4751) open CLIENTS/CLIENT6/~DMTMP/WORDPRO/BENCHS.LWP failed for handle 4330 (File exists)
(4752) nb_read: handle 4330 was not open size=24 ofs=2156
(4754) nb_read: handle 4330 was not open size=276 ofs=2139
(4755) nb_read: handle 4330 was not open size=14 ofs=2136
(4757) nb_read: handle 4330 was not open size=7 ofs=2136
(4759) nb_read: handle 4330 was not open size=12 ofs=2138
(4761) nb_read: handle 4330 was not open size=8 ofs=2137
(4763) nb_read: handle 4330 was not open size=15 ofs=2135
(4765) nb_read: handle 4330 was not open size=16 ofs=0
(4767) nb_read: handle 4330 was not open size=2048 ofs=16
(4768) nb_read: handle 4330 was not open size=2048 ofs=2064
(4769) nb_read: handle 4330 was not open size=2048 ofs=257
(4770) nb_read: handle 4330 was not open size=2048 ofs=385
(4771) nb_read: handle 4330 was not open size=2048 ofs=513
(4772) nb_read: handle 4330 was not open size=2048 ofs=641
(4773) nb_read: handle 4330 was not open size=2048 ofs=769
(4774) nb_read: handle 4330 was not open size=2048 ofs=897
(4775) nb_read: handle 4330 was not open size=2048 ofs=1025
(4776) nb_read: handle 4330 was not open size=2048 ofs=1153
(4777) nb_read: handle 4330 was not open size=2048 ofs=1281
(4778) nb_read: handle 4330 was not open size=2048 ofs=1409
(4779) nb_read: handle 4330 was not open size=408 ofs=1537
..(4899) nb_close: handle 4330 was not open
(4964) nb_read: handle 4329 was not open size=1386 ofs=2157
(4971) nb_read: handle 4329 was not open size=4096 ofs=2157
(4972) nb_read: handle 4329 was not open size=4096 ofs=6253
(4973) nb_read: handle 4329 was not open size=4096 ofs=10349
(4974) nb_read: handle 4329 was not open size=4096 ofs=14445
(4975) nb_read: handle 4329 was not open size=4096 ofs=18541
(4976) nb_read: handle 4329 was not open size=256 ofs=22637
(4977) nb_read: handle 4329 was not open size=4096 ofs=22893
(4978) nb_read: handle 4329 was not open size=4096 ofs=26989
(4979) nb_read: handle 4329 was not open size=4096 ofs=31085
(4980) nb_read: handle 4329 was not open size=4096 ofs=35181
(4981) nb_read: handle 4329 was not open size=4096 ofs=39277
(4982) nb_read: handle 4329 was not open size=4096 ofs=43373
(4983) nb_read: handle 4329 was not open size=4096 ofs=47469
(4986) nb_read: handle 4329 was not open size=1024 ofs=2157
(4987) nb_read: handle 4329 was not open size=4688 ofs=1667
(4988) nb_read: handle 4329 was not open size=3516 ofs=6355
(4989) nb_read: handle 4329 was not open size=16384 ofs=9871
(4990) nb_read: handle 4329 was not open size=3771 ofs=26255
(4991) nb_read: handle 4329 was not open size=2048 ofs=30026
(4992) nb_read: handle 4329 was not open size=1024 ofs=32074
(4993) nb_read: handle 4329 was not open size=6652 ofs=33098
(4994) nb_read: handle 4329 was not open size=16384 ofs=39750
(4995) nb_read: handle 4329 was not open size=10496 ofs=56134
(4996) nb_read: handle 4329 was not open size=1784 ofs=66630
(4997) nb_read: handle 4329 was not open size=90 ofs=68414
(4998) nb_read: handle 4329 was not open size=273 ofs=68504
(4999) nb_read: handle 4329 was not open size=1600 ofs=68777
(5000) nb_read: handle 4329 was not open size=640 ofs=70377
(5001) nb_read: handle 4329 was not open size=1280 ofs=71017
(5002) nb_read: handle 4329 was not open size=8464 ofs=72297
(5003) nb_read: handle 4329 was not open size=158 ofs=80761
(5004) nb_read: handle 4329 was not open size=912 ofs=80919
(5005) nb_read: handle 4329 was not open size=1980 ofs=81831
(5006) nb_read: handle 4329 was not open size=2256 ofs=83811
(5007) nb_read: handle 4329 was not open size=400 ofs=86067
(5008) nb_read: handle 4329 was not open size=252 ofs=86467
(5009) nb_read: handle 4329 was not open size=42 ofs=86719
(5011) nb_read: handle 4329 was not open size=725 ofs=86761
(5012) nb_read: handle 4329 was not open size=512 ofs=2009
(5013) nb_read: handle 4329 was not open size=9 ofs=3895
(5015) nb_read: handle 4329 was not open size=46 ofs=3781
(5017) nb_read: handle 4329 was not open size=92 ofs=4059
(5018) nb_read: handle 4329 was not open size=92 ofs=3781
(5019) nb_read: handle 4329 was not open size=184 ofs=3986
(5020) nb_read: handle 4329 was not open size=16384 ofs=3987
(5021) nb_read: handle 4329 was not open size=16384 ofs=3902
(5022) nb_read: handle 4329 was not open size=10240 ofs=3781
(5023) nb_read: handle 4329 was not open size=16384 ofs=3821
(5024) nb_read: handle 4329 was not open size=935 ofs=3891
(5025) nb_read: handle 4329 was not open size=10240 ofs=4060
(5026) nb_read: handle 4329 was not open size=184 ofs=4059
(5027) nb_read: handle 4329 was not open size=184 ofs=256
(5028) nb_read: handle 4329 was not open size=1026 ofs=256
(5029) nb_read: handle 4329 was not open size=326 ofs=3901
(5030) nb_read: handle 4329 was not open size=652 ofs=3899
(5031) nb_read: handle 4329 was not open size=2 ofs=256
(5033) nb_read: handle 4329 was not open size=4948 ofs=258
(5034) nb_close: handle 4329 was not open
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+...+......................................................................+..................................+.............................+.....................+++.........................................................................................................................+....................+...............+.....................................+......+......+.+...............................+.+....+.......................+.................+..................++....+.+..+..........++..+...........+.........+.........+..........................++++........+++++.............................+.++...........+..++++++**************************************************
Throughput 27.8199 MB/sec (NB=34.7749 MB/sec 278.199 MBit/sec)
Average latency 6.940010 ms
Maximum latency 18993.291000 ms
Latency > 2 sec. 0.081048 %
Latency > 10 sec. 0.015581 %
50 clients started
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+.................................................................+....................................................................................................................................................+..................+...+.............................................................................................................................................................................................................+..................+.......................................................+................++.......+............+.....+...............................+..+...............++.+.................................+............+.....++..++.......................+.........+...............++...........................+++.........+++.+...............+....+...+.....+........+......++....++++++.++**************************************************
Throughput 27.0223 MB/sec (NB=33.7779 MB/sec 270.223 MBit/sec)
Average latency 6.739319 ms
Maximum latency 18699.530000 ms
Latency > 2 sec. 0.090834 %
Latency > 10 sec. 0.013650 %
50 clients started
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................................................+............................................................................+...................................................................................................................................................................+.+................................................................+.................................................++++...........................+..+....+..........++...........+....................+.....+.............................................+.............+................+...+..........+++....+....+............................+........+..+.....+....+...+........+........+++...............+++++..++++.++++**************************************************
Throughput 27.2861 MB/sec (NB=34.1077 MB/sec 272.861 MBit/sec)
Average latency 7.051013 ms
Maximum latency 23801.490000 ms
Latency > 2 sec. 0.091263 %
Latency > 10 sec. 0.014722 %
50 clients started
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
[root@inif w]# umount /mnt
[root@inif w]# e2fsck -f -y /dev/sdc
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -(1250434--1250435) -(1273360--1273370)
Fix? yes
Free blocks count wrong for group #38 (32253, counted=32266). Fix? yes
Free blocks count wrong (1003250, counted=1003263). Fix? yes
/dev/sdc: ***** FILE SYSTEM WAS MODIFIED *****/dev/sdc: 68842/640000 files (0.1% non-contiguous), 276737/1280000 blocks
Nov 1 16:27:35 inif kernel: kjournald starting. Commit interval 5 seconds Nov 1 16:27:35 inif kernel: EXT3 FS on sdc, internal journal Nov 1 16:27:35 inif kernel: EXT3-fs: mounted filesystem with ordered data mode. Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:29:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:29:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:30:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9376352 Nov 1 16:30:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:30:19 inif kernel: end_request: I/O error, dev sdc, sector 9995808 Nov 1 16:30:19 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #608098 offset 0 Nov 1 16:30:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:31:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:31:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:31:05 inif kernel: end_request: I/O error, dev sdc, sector 367432 Nov 1 16:31:05 inif kernel: __ratelimit: 62 callbacks suppressed Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45929 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45930 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45931 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45932 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45933 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45934 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45935 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45936 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45937 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:05 inif kernel: Buffer I/O error on device sdc, logical block 45938 Nov 1 16:31:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:31:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:31:22 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:31:54 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:31:59 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:32:40 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:32:45 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:33:26 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:33:31 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:34:12 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 9621088 Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 10145456 Nov 1 16:34:13 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:13 inif kernel: end_request: I/O error, dev sdc, sector 7095792 Nov 1 16:34:17 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:34:58 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9530232 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798712 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798728 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798736 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 10059240 Nov 1 16:34:59 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:34:59 inif kernel: end_request: I/O error, dev sdc, sector 9798720 Nov 1 16:35:03 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:35:44 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:35:49 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:36:30 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:36:31 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:36:31 inif kernel: end_request: I/O error, dev sdc, sector 10032752 Nov 1 16:36:35 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:37:16 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8862288 Nov 1 16:37:17 inif kernel: __ratelimit: 59 callbacks suppressed Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107786 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107787 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107788 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107789 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107790 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107791 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107792 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107793 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107794 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: Buffer I/O error on device sdc, logical block 1107795 Nov 1 16:37:17 inif kernel: lost page write due to I/O error on sdc Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8892560 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8893456 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8894480 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8895512 Nov 1 16:37:17 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:37:17 inif kernel: end_request: I/O error, dev sdc, sector 8896536 Nov 1 16:37:21 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:37:29 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:38:02 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9213944 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280464 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280472 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9280480 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9184784 Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560038 offset 0 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10159136 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 303616 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9586656 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 9641352 Nov 1 16:38:03 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:38:03 inif kernel: end_request: I/O error, dev sdc, sector 10061776 Nov 1 16:38:03 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #576172 offset 0 Nov 1 16:38:07 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:38:48 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:38:53 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:39:34 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:39:39 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:40:20 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:40:25 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:41:06 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:41:11 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:41:52 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:41:53 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:41:53 inif kernel: end_request: I/O error, dev sdc, sector 647544 Nov 1 16:41:53 inif kernel: __ratelimit: 639 callbacks suppressed Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80943 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80944 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80945 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80946 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80947 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80948 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80949 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80950 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80951 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:53 inif kernel: Buffer I/O error on device sdc, logical block 80952 Nov 1 16:41:53 inif kernel: lost page write due to I/O error on sdc Nov 1 16:41:57 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:42:05 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:42:38 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:42:43 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:43:24 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:43:25 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:43:25 inif kernel: end_request: I/O error, dev sdc, sector 478480 Nov 1 16:43:25 inif kernel: __ratelimit: 118 callbacks suppressed Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59810 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59811 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59812 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59813 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59814 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59815 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59816 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59817 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59818 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:25 inif kernel: Buffer I/O error on device sdc, logical block 59819 Nov 1 16:43:25 inif kernel: lost page write due to I/O error on sdc Nov 1 16:43:29 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:44:10 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:44:15 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:44:56 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:44:57 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:44:57 inif kernel: end_request: I/O error, dev sdc, sector 9353840 Nov 1 16:44:57 inif kernel: __ratelimit: 44 callbacks suppressed Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169230 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169231 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169232 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169233 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169234 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169235 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169236 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169237 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169238 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:44:57 inif kernel: Buffer I/O error on device sdc, logical block 1169239 Nov 1 16:44:57 inif kernel: lost page write due to I/O error on sdc Nov 1 16:45:01 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:45:11 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:45:42 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:45:47 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:46:28 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9226672 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9426536 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9832040 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9895976 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9896048 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9946768 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 279576 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 1048576 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 7097712 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9102152 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10190752 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10199216 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 295416 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9057504 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115608 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9289416 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9529432 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9577376 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9672304 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9763984 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9789792 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9811504 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9826536 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9115656 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9624224 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9928144 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10003472 Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=517632, block=1250434 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 10186880 Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_free_branches: Read failure, inode=560474, block=1273360 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 332944 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 451768 Nov 1 16:46:29 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:46:29 inif kernel: end_request: I/O error, dev sdc, sector 9208392 Nov 1 16:46:29 inif kernel: EXT3-fs error (device sdc): ext3_find_entry: reading directory #560267 offset 0 Nov 1 16:46:33 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:47:14 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 429128 Nov 1 16:47:15 inif kernel: __ratelimit: 67 callbacks suppressed Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 53641 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 10138096 Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267262 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267263 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267264 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267265 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267266 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267267 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267268 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267269 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: Buffer I/O error on device sdc, logical block 1267270 Nov 1 16:47:15 inif kernel: lost page write due to I/O error on sdc Nov 1 16:47:15 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:47:15 inif kernel: end_request: I/O error, dev sdc, sector 964064 Nov 1 16:47:19 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:47:28 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:47:33 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:48:00 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:48:05 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:48:46 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10082504 Nov 1 16:48:47 inif kernel: __ratelimit: 164 callbacks suppressed Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260313 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260314 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260315 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260316 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260317 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260318 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260319 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260320 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260321 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: Buffer I/O error on device sdc, logical block 1260322 Nov 1 16:48:47 inif kernel: lost page write due to I/O error on sdc Nov 1 16:48:47 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:48:47 inif kernel: end_request: I/O error, dev sdc, sector 10086400 Nov 1 16:48:51 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:49:00 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:49:32 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:49:33 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:49:33 inif kernel: end_request: I/O error, dev sdc, sector 9777376 Nov 1 16:49:33 inif kernel: __ratelimit: 132 callbacks suppressed Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222172 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222173 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222174 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222175 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222176 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222177 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222178 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222179 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222180 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:33 inif kernel: Buffer I/O error on device sdc, logical block 1222181 Nov 1 16:49:33 inif kernel: lost page write due to I/O error on sdc Nov 1 16:49:37 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:49:46 inif kernel: JBD: Detected IO errors while flushing file data on sdc Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:50:18 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9728192 Nov 1 16:50:19 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:50:19 inif kernel: end_request: I/O error, dev sdc, sector 9350872 Nov 1 16:50:23 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: scsi(20:0:0): LOOP RESET ISSUED. Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP reset occured (0). Nov 1 16:51:04 inif kernel: qla2xxx 0000:04:02.0: LIP occured (0). Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709760 Nov 1 16:51:05 inif kernel: __ratelimit: 54 callbacks suppressed Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213720 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213721 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213722 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213723 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213724 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213725 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213726 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213727 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213728 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: Buffer I/O error on device sdc, logical block 1213729 Nov 1 16:51:05 inif kernel: lost page write due to I/O error on sdc Nov 1 16:51:05 inif kernel: sd 16:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK,SUGGEST_OK Nov 1 16:51:05 inif kernel: end_request: I/O error, dev sdc, sector 9709864 Nov 1 16:51:09 inif kernel: qla2xxx 0000:04:02.0: qla2xxx_eh_bus_reset: reset succeded
Quoted 23 lines · Vladislav Bolkhovitin
Quoted 22 lines · Nick Piggin
Quoted 21 lines · Vladislav Bolkhovitin
Quoted 19 lines · Nick Piggin
Quoted 7 lines · Vladislav Bolkhovitin
Hi,
During recent debugging session of my SCSI target SCST
(http://scst.sf.net) I noticed manyWARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x51/0x66()Quoted 7 lines
messages in kernel log on the initiator. I attached the full log of several of them.
My target was buggy and I was working on fixing it, but I suppose Linux should handle such failures more gracefully. In all the cases the target had one type of failure: it "ate" a SCSI command and never returned result of it.
Right. This is one of the warnings I see in my fault-injection testing. It is fixed by my patch to clean up and improve the page and buffer error handling in the vm/fs.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Can you specify which patch you referring? Is it in 2.6.27?
It's just an RFC at the moment which I posted to fsdevel. Not in 2.6.27.
I see. I'm looking forward to see it in 2.6.28 or .29. This is really a needed work.
Hopefully. Unfortunately it doesn't exactly make the filesystems themselves more robust against failure. That needs to be done on a case by case basis.
Quoted 9 lines
BTW, have you even seen in your fault-injection testing that after receiving a failure from a SCSI device during heavy load ext3 file system mounted on it gets corrupted and journal replay on remount doesn't repair it, only manual e2fsck helps? I've many times seen that, including cases when the target was remaining up and fully functional. See, e.g., "MOANING MODE ON" part in
http://marc.info/?l=linux-scsi&m=121932252324432&w=2. I haven't checked that case since then, although I see such corruptions quite often. But in all them I can't so clearly say that it isn't a target's failure.
I haven't seen that, but I'm not exactly testing for filesystem robustness to errors, but more of core vm/fs layer robustness, so I'm mainly trying to inject errors into data portion of inodes.
I think the ext3 development list should be interested in your report.
Thanks,
Nick