MD: Quickly return errors if too many devices have failed.
PATCH 1/1 MD: Quickly return errors if too many devices have failed.
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3
Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1
Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null)
(null) (null) 1
Mar 12 11:52:53 bp-01 kernel: ------------[ cut here ]------------
Mar 12 11:52:53 bp-01 kernel: WARNING: at drivers/md/raid5.c:354 init_stripe+0x2d4/0x370 [raid456]()
Mar 12 11:52:53 bp-01 kernel: Hardware name: PowerEdge R415
Mar 12 11:52:53 bp-01 kernel: Modules linked in: raid456 md_mod async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx sunrpc ipv6 dcdbas freq_table mperf kvm_amd kvm crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw fam15h_power k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core bnx2 sg ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul pata_acpi ata_generic pata_atiixp ahci libahci mptsas mptscsih mptbase scsi_transport_sas bfa scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
Mar 12 11:52:53 bp-01 kernel: Pid: 8604, comm: dd Not tainted 3.8.0 #8
Mar 12 11:52:53 bp-01 kernel: Call Trace:
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81056dcf>] warn_slowpath_common+0x7f/0xc0
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81056e2a>] warn_slowpath_null+0x1a/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa040f854>] init_stripe+0x2d4/0x370 [raid456]
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa040fc45>] get_active_stripe+0x355/0x3f0 [raid456]
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8107c050>] ? wake_up_bit+0x40/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa0413c66>] make_request+0x1a6/0x430 [raid456]
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8107c050>] ? wake_up_bit+0x40/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffffa03f0123>] md_make_request+0xd3/0x200 [md_mod]
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81122cd5>] ? mempool_alloc_slab+0x15/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81122e40>] ? mempool_alloc+0x60/0x170
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8124a8ba>] generic_make_request+0xca/0x100
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8124a969>] submit_bio+0x79/0x160
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b4825>] ? bio_alloc_bioset+0x65/0x120
Mar 12 11:52:53 bp-01 kernel: [<ffffffff815458b2>] ? _raw_spin_unlock_irqrestore+0x12/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811af318>] submit_bh+0x128/0x200
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b1dd0>] __block_write_full_page+0x1e0/0x330
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b06b0>] ? lock_buffer+0x30/0x30
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b5920>] ? I_BDEV+0x10/0x10
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b5920>] ? I_BDEV+0x10/0x10
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b2055>] block_write_full_page+0x15/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6988>] blkdev_writepage+0x18/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112a5f7>] __writepage+0x17/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112b875>] write_cache_pages+0x245/0x520
Mar 12 11:52:53 bp-01 kernel: [<ffffffff810844c9>] ? __wake_up_common+0x59/0x90
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112a5e0>] ? set_page_dirty+0x60/0x60
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112bba1>] generic_writepages+0x51/0x80
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811bd8bf>] ? send_to_group+0x13f/0x200
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112bbf0>] do_writepages+0x20/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811210f1>] __filemap_fdatawrite_range+0x51/0x60
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8112133f>] filemap_fdatawrite+0x1f/0x30
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81121385>] filemap_write_and_wait+0x35/0x60
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6ca1>] __sync_blockdev+0x21/0x40
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6cd3>] sync_blockdev+0x13/0x20
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b6ed9>] __blkdev_put+0x69/0x1d0
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b7096>] blkdev_put+0x56/0x140
Mar 12 11:52:53 bp-01 kernel: [<ffffffff811b71a4>] blkdev_close+0x24/0x30
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8117f7bb>] __fput+0xbb/0x260
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8117f9ce>] ____fput+0xe/0x10
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81077e5f>] task_work_run+0x8f/0xf0
Mar 12 11:52:53 bp-01 kernel: [<ffffffff81014a54>] do_notify_resume+0x84/0x90
Mar 12 11:52:53 bp-01 kernel: [<ffffffff8154e992>] int_signal+0x12/0x17
Mar 12 11:52:53 bp-01 kernel: ---[ end trace d71de83816e8d215 ]---Are other people seeing this, or is this an artifact of the way I am killing devices ('echo offline > /sys/block/$dev/device/state')?
I would prefer to get immediate errors if nothing can be done to satisfy the request and I've been thinking of something like the attached patch. The patch below is incomplete. It does not take into account any reshaping that is going on, nor does it try to figure out if a mirror set in RAID10 has died; but I hope it gets the basic idea across.
Is this a good way to handle this situation, or am I missing something?
brassow
<no comments: POC>
drivers/md/raid1.c +16 -0
@@ -210,6 +210,17 @@ static void put_buf(struct r1bio *r1_biolower_barrier (conf); } +static int has_failed (struct r1conf *conf) { +struct mddev *mddev = conf->mddev; +struct md_rdev *rdev; + +rdev_for_each (rdev, mddev) +if (likely (!test_bit (Faulty, &rdev->flags))) +return 0 ; + +return 1 ; +} +static void reschedule_retry (struct r1bio *r1_bio) {unsigned long flags;
@@ -1007,6 +1018,11 @@ static void make_request(struct mddev *mint sectors_handled;int max_sectors; +if (has_failed (conf)) { +bio_endio (bio, -EIO ); +return ; + } +/* * Register the new request and wait if the reconstruction * thread has put up a bar for new requests.
drivers/md/raid10.c +17 -0
@@ -270,6 +270,18 @@ static void put_buf(struct r10bio *r10_blower_barrier (conf); } +static int has_failed (struct r10conf *conf) { +struct mddev *mddev = conf->mddev; +struct md_rdev *rdev; + +rdev_for_each (rdev, mddev) +if (likely (!test_bit (Faulty, &rdev->flags))) +return 0 ; + +return 1 ; + +} +static void reschedule_retry (struct r10bio *r10_bio) {unsigned long flags;
@@ -1159,6 +1171,11 @@ static void make_request(struct mddev *mint max_sectors;int sectors; +if (has_failed (conf)) { +bio_endio (bio, -EIO ); +return ; + } +if (unlikely (bio->bi_rw &REQ_FLUSH )) {md_flush_request (mddev, bio);return ;
drivers/md/raid5.c +5 -0
@@ -4241,6 +4241,11 @@ static void make_request(struct mddev *mconst int rw =bio_data_dir (bi);int remaining; +if (has_failed (conf)) { +bio_endio (bi, -EIO ); +return ; + } +if (unlikely (bi->bi_rw &REQ_FLUSH )) {md_flush_request (mddev, bi);return ;
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1
This is the third report in as many weeks that mentions that WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5: schedule_construction should abort if nothing to do.
which is commit 29d90fa2adbdd9f in linux-next.
Quoted 6 lines
Mar 12 11:52:53 bp-01 kernel: ------------[ cut here ]------------ Mar 12 11:52:53 bp-01 kernel: WARNING: at drivers/md/raid5.c:354 init_stripe+0x2d4/0x370 [raid456]()Are other people seeing this, or is this an artifact of the way I am killing devices ('echo offline >
/sys/block/$dev/device/state')?
That is a perfectly good way to kill a device.
Quoted 8 lines
I would prefer to get immediate errors if nothing can be done to satisfy the request and I've been thinking of something like the attached patch. The patch below is incomplete. It does not take into account any reshaping that is going on, nor does it try to figure out if a mirror set in RAID10 has died; but I hope it gets the basic idea across.
Is this a good way to handle this situation, or am I missing something?
I think we do get immediate errors (once all bugs are fixed). Your patch does extra work for every request which is only of value if the array has failed - and it really doesn't make sense to optimise for a failed array.
The current approach is to just try to satisfy a request and once we find that we need to do something that is impossible - return an error at that point. I think that is best.
Can you try the commit I identified and see if it makes the problem go away?
Thanks,
NeilBrown
Quoted 18 lines · NeilBrown
Quoted 9 lines
I would prefer to get immediate errors if nothing can be done to satisfy the request and I've been thinking of something like the attached patch. The patch below is incomplete. It does not take into account any reshaping that is going on, nor does it try to figure out if a mirror set in RAID10 has died; but I hope it gets the basic idea across.
Is this a good way to handle this situation, or am I missing something?
I think we do get immediate errors (once all bugs are fixed). Your patch does extra work for every request which is only of value if the array has failed - and it really doesn't make sense to optimise for a failed array.
The current approach is to just try to satisfy a request and once we find that we need to do something that is impossible - return an error at that point. I think that is best.
Quoted 1 line
Can you try the commit I identified and see if it makes the problem go away?
Yes, that sounds better. I will try with the commit you mentioned.
thanks,
brassow
Quoted 4 lines
Quoted 2 lines
Are other people seeing this, or is this an artifact of the way I am killing devices ('echo offline >
/sys/block/$dev/device/state')?That is a perfectly good way to kill a device.
Anyone that knows what may have replaced this in 3.8, or if this isn't available for virtio devices? I have a vm on 3.8 (ubuntu raring alpha/beta/something), and I find no such file there. Works great on the host OS, though (ubuntu precise, linux 3.2).
Vennlige hilsener / Best regards
roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Roy Sigurd Karlsbakk
Quoted 4 lines
Quoted 2 lines
Are other people seeing this, or is this an artifact of the way I am killing devices ('echo offline >
/sys/block/$dev/device/state')?That is a perfectly good way to kill a device.
Anyone that knows what may have replaced this in 3.8, or if this isn't available for virtio devices? I have a vm on 3.8 (ubuntu raring alpha/beta/something), and I find no such file there. Works great on the host OS, though (ubuntu precise, linux 3.2).
I'm using this specifically on 3.8+. I'm not using VMs at the moment, but in the past it has been available there - not for partitions and loopback devices though.
brassow
Quoted 12 lines · NeilBrown
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1
Quoted 7 lines
This is the third report in as many weeks that mentions that
WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5:schedule_constructionshould abort if nothing to do.which is commit
29d90fa2adbdd9fin linux-next.
Sorry, I don't see this commit in linux-next:
(the "for-next" branch of) git://github.com/neilbrown/linux.git
or git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
Where should I be looking?
I did grab a patch from an earlier discussion where you mentioned a similar commit ID. It didn't solve the problem, but it did prevent an endless progression of the same error messages. I only saw one instance of the above after the patch.
I'm fairly certain that the hang was affecting more than just RAID5 though. It also happened with raid1/10. I'll go back with 3.9.0-rc3 and make sure that's true until I can figure out which 'linux-next' commit you are talking about.
thanks,
brassow
Quoted 21 lines · On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 18 lines · NeilBrown
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1This is the third report in as many weeks that mentions that
WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5:schedule_constructionshould abort if nothing to do.
Quoted 7 lines
Quoted 1 line
which is commit
29d90fa2adbdd9fin linux-next.Sorry, I don't see this commit in linux-next:
(the "for-next" branch of)git://github.com/neilbrown/linux.git
orgit://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitWhere should I be looking?
Sorry, I probably messed up.
I meant this commit:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca
Quoted 4 lines
I did grab a patch from an earlier discussion where you mentioned a similar commit ID. It didn't solve the problem, but it did prevent an endless progression of the same error messages. I only saw one instance of the above after the patch.
I'm fairly certain that the hang was affecting more than just RAID5 though. It also happened with raid1/10. I'll go back with 3.9.0-rc3 and make sure that's true until I can figure out which 'linux-next' commit you are talking about.
If you could get something concrete I'd love to hear about it, thanks.
NeilBrown
Quoted 30 lines · NeilBrown
Quoted 27 lines · On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 12 lines · NeilBrown
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1Quoted 7 lines
This is the third report in as many weeks that mentions that
WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5:schedule_constructionshould abort if nothing to do.which is commit
29d90fa2adbdd9fin linux-next.Sorry, I don't see this commit in linux-next:
(the "for-next" branch of)git://github.com/neilbrown/linux.git
orgit://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
Quoted 5 lines
Quoted 1 line
Where should I be looking?
Sorry, I probably messed up.
I meant this commit:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8ca
Yes, I found this patch in 'for-next'. I tested 3.9.0-rc3 with and without this patch. The good news is that my issue with RAID5 appears to be fixed with this patch. To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size). Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write. After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write. That's good. Will this patch make it back to 3.[78]?
However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min. Hung task messages with call traces and many many errors are the result. This is bad. I haven't figured out why these are so slow yet.
On a different topic, I've noticed the following commits in 'for-next':
90584fc MD: Prevent sysfs operations on uninitialized kobjects
e3620a3 MD RAID5: Avoid accessing gendisk or queue structs when not available
but these are not in 3.9.0-rc3. They should make their way into 3.9.0 as well as 3.8.0. (They apply cleanly to the 3.8 kernel, but I hadn't bothered to notify 'stable' - only mention the regression was introduced in 3.8-rc1.)
Thanks,
brassow
Quoted 35 lines · On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 32 lines · NeilBrown
Quoted 28 lines · On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 12 lines · NeilBrown
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1Quoted 7 lines
This is the third report in as many weeks that mentions that
WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5:schedule_constructionshould abort if nothing to do.which is commit
29d90fa2adbdd9fin linux-next.Sorry, I don't see this commit in linux-next:
(the "for-next" branch of)git://github.com/neilbrown/linux.git
orgit://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitWhere should I be looking?
Quoted 7 lines
Quoted 3 lines
Sorry, I probably messed up.
I meant this commit:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8caYes, I found this patch in 'for-next'. I tested 3.9.0-rc3 with and without this patch. The good news is that my issue with RAID5 appears to be fixed with this patch. To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size). Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write. After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write. That's good. Will this patch make it back to 3.[78]?
However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min. Hung task messages with call traces and many many errors are the result. This is bad. I haven't figured out why these are so slow yet.
What happens if you take RAID out of the picture?
i.e. write to a single device, then "kill" that device, then try issuing a 40M write request to it.
If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to also take 2.5 minutes to resolve.
If it resolves much more quickly than it does with RAID1, then that is a problem we should certainly address.
Quoted 5 lines
On a different topic, I've noticed the following commits in 'for-next':
90584fc MD: Prevent sysfs operations on uninitialized kobjects
e3620a3 MD RAID5: Avoid accessing gendisk or queue structs when not available
but these are not in 3.9.0-rc3. They should make their way into 3.9.0 as well as 3.8.0. (They apply cleanly to the 3.8 kernel, but I hadn't bothered to notify 'stable' - only mention the regression was introduced in 3.8-rc1.)
They are due to be sent to Linus today.
The second one is tagged for -stable and will go to 3.8.x (I think 3.7.x is closed).
The first isn't - my quick examination suggested that the current code is safe but not ideal. If you think it is appropriate for -stable (i.e. fixes a real bug that could hurt users) let me know why and I'll make sure it goes to -stable.
NeilBrown
Quoted 46 lines · NeilBrown
Quoted 35 lines · On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 32 lines · NeilBrown
Quoted 28 lines · On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 12 lines · NeilBrown
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1Quoted 7 lines
This is the third report in as many weeks that mentions that
WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5:schedule_constructionshould abort if nothing to do.which is commit
29d90fa2adbdd9fin linux-next.Sorry, I don't see this commit in linux-next:
(the "for-next" branch of)git://github.com/neilbrown/linux.git
orgit://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitWhere should I be looking?
Quoted 7 lines
Quoted 3 lines
Sorry, I probably messed up.
I meant this commit:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8caYes, I found this patch in 'for-next'. I tested 3.9.0-rc3 with and without this patch. The good news is that my issue with RAID5 appears to be fixed with this patch. To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size). Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write. After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write. That's good. Will this patch make it back to 3.[78]?
However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min. Hung task messages with call traces and many many errors are the result. This is bad. I haven't figured out why these are so slow yet.
Quoted 8 lines
What happens if you take RAID out of the picture?
i.e. write to a single device, then "kill" that device, then try issuing a 40M write request to it.If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to also take 2.5 minutes to resolve.
If it resolves much more quickly than it does with RAID1, then that is a problem we should certainly address.
The test is a little different because once you offline a device, you can't open it. So, I had to start I/O and then kill the device. I still get 158MB/s - 3 orders of magnitude faster than RAID1. Besides, if RAID10 takes 9+ minutes to complete, we'd still have something to fix. I have also tested this with an "error" device and it also returns in sub-second time.
brassow
[root@bp-01 ~]# off.sh sda
Turning off sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda1 bs=4M count=10
dd: opening `/dev/sda1': No such device or address [root@bp-01 ~]# on.sh sda
Turning on sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda bs=4M count=1000 &
[1] 5203
[root@bp-01 ~]# off.sh sda
Turning off sda
[root@bp-01 ~]# 1000+0 records in
1000+0 records out
4194304000 bytes (4.2 GB) copied, 26.5564 s, 158 MB/s
Quoted 62 lines · On Thu, 21 Mar 2013 08:58:54 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 46 lines · NeilBrown
Quoted 35 lines · On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 32 lines · NeilBrown
Quoted 28 lines · On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 12 lines · NeilBrown
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1Quoted 7 lines
This is the third report in as many weeks that mentions that
WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5:schedule_constructionshould abort if nothing to do.which is commit
29d90fa2adbdd9fin linux-next.Sorry, I don't see this commit in linux-next:
(the "for-next" branch of)git://github.com/neilbrown/linux.git
orgit://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitWhere should I be looking?
Quoted 7 lines
Quoted 3 lines
Sorry, I probably messed up.
I meant this commit:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8caYes, I found this patch in 'for-next'. I tested 3.9.0-rc3 with and without this patch. The good news is that my issue with RAID5 appears to be fixed with this patch. To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size). Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write. After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write. That's good. Will this patch make it back to 3.[78]?
However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min. Hung task messages with call traces and many many errors are the result. This is bad. I haven't figured out why these are so slow yet.
Quoted 8 lines
What happens if you take RAID out of the picture?
i.e. write to a single device, then "kill" that device, then try issuing a 40M write request to it.If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to also take 2.5 minutes to resolve.
If it resolves much more quickly than it does with RAID1, then that is a problem we should certainly address.The test is a little different because once you offline a device, you can't open it. So, I had to start I/O and then kill the device. I still get 158MB/s - 3 orders of magnitude faster than RAID1. Besides, if RAID10 takes 9+ minutes to complete, we'd still have something to fix. I have also tested this with an "error" device and it also returns in sub-second time.
brassow
Quoted 14 lines
[root@bp-01 ~]#
off.shsda
Turning off sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda1 bs=4M count=10
dd: opening `/dev/sda1': No such device or address [root@bp-01 ~]#on.shsda
Turning on sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda bs=4M count=1000 &
[1] 5203
[root@bp-01 ~]#off.shsda
Turning off sda
[root@bp-01 ~]# 1000+0 records in
1000+0 records out
4194304000 bytes (4.2 GB) copied, 26.5564 s, 158 MB/s
Maybe if you could show me some/all of the error messages that you get during these long delays it might help. Also the error messages you (presumably) got from the kernel from the above plain-disk test.
It should quickly fail all but one copy of the data, then try writing to that copy exactly the same way that it would write to a plain disk.
For RAID10 large writes have to be chopped up for striping, so the extra requests which all have to fail could be the reason for the extra delay with RAID10.
NeilBrown
Quoted 84 lines · NeilBrown
Quoted 62 lines · On Thu, 21 Mar 2013 08:58:54 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 46 lines · NeilBrown
Quoted 35 lines · On Wed, 20 Mar 2013 15:56:03 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 32 lines · NeilBrown
Quoted 28 lines · On Tue, 19 Mar 2013 16:15:35 -0500 Brassow Jonathan <[email protected]> wrote:
Quoted 12 lines · NeilBrown
Quoted 8 lines · On Wed, 13 Mar 2013 12:29:24 -0500 Jonathan Brassow <[email protected]> wrote:
Neil,
I've noticed that when too many devices fail in a RAID arrary that addtional I/O will hang, yielding an endless supply of:
Mar 12 11:52:53 bp-01 kernel: Buffer I/O error on device md1, logical block 3 Mar 12 11:52:53 bp-01 kernel: lost page write due to I/O error on md1 Mar 12 11:52:53 bp-01 kernel: sector=800 i=3 (null) (null) (null) (null) 1Quoted 7 lines
This is the third report in as many weeks that mentions that
WARN_ON. The first two where quite different causes.
I think this one is the same as the first one, which means it would be fixed by
md/raid5:schedule_constructionshould abort if nothing to do.which is commit
29d90fa2adbdd9fin linux-next.Sorry, I don't see this commit in linux-next:
(the "for-next" branch of)git://github.com/neilbrown/linux.git
orgit://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.gitWhere should I be looking?
Quoted 7 lines
Quoted 3 lines
Sorry, I probably messed up.
I meant this commit:
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=ce7d363aaf1e28be8406a2976220944ca487e8caYes, I found this patch in 'for-next'. I tested 3.9.0-rc3 with and without this patch. The good news is that my issue with RAID5 appears to be fixed with this patch. To test, I simply created a 1GB RAID array, let it sync, killed all of the devices and then issued a 40M write request (4M block size). Before the patch, I would see the kernel warnings and it would take 7+ minutes to finish the 40M write. After the patch, I don't see the kernel warnings or call traces and it takes < 1 sec to finish the 40M write. That's good. Will this patch make it back to 3.[78]?
However, I also found that RAID1 can take 2.5 min to perform the write and RAID10 can take 9+ min. Hung task messages with call traces and many many errors are the result. This is bad. I haven't figured out why these are so slow yet.
Quoted 8 lines
What happens if you take RAID out of the picture?
i.e. write to a single device, then "kill" that device, then try issuing a 40M write request to it.If that takes 2.5 minutes to resolve, then I think it is correct for RAID1 to also take 2.5 minutes to resolve.
If it resolves much more quickly than it does with RAID1, then that is a problem we should certainly address.The test is a little different because once you offline a device, you can't open it. So, I had to start I/O and then kill the device. I still get 158MB/s - 3 orders of magnitude faster than RAID1. Besides, if RAID10 takes 9+ minutes to complete, we'd still have something to fix. I have also tested this with an "error" device and it also returns in sub-second time.
brassow
Quoted 14 lines
[root@bp-01 ~]#
off.shsda
Turning off sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda1 bs=4M count=10
dd: opening `/dev/sda1': No such device or address [root@bp-01 ~]#on.shsda
Turning on sda
[root@bp-01 ~]# dd if=/dev/zero of=/dev/sda bs=4M count=1000 &
[1] 5203
[root@bp-01 ~]#off.shsda
Turning off sda
[root@bp-01 ~]# 1000+0 records in
1000+0 records out
4194304000 bytes (4.2 GB) copied, 26.5564 s, 158 MB/sMaybe if you could show me some/all of the error messages that you get during these long delays it might help. Also the error messages you (presumably) got from the kernel from the above plain-disk test.
Quoted 6 lines
It should quickly fail all but one copy of the data, then try writing to that copy exactly the same way that it would write to a plain disk.
For RAID10 large writes have to be chopped up for striping, so the extra requests which all have to fail could be the reason for the extra delay with RAID10.
Found the problem. dm-raid does not support badblock tracking. dm-raid was not setting badblocks.shift = -1. Thus, RAID1/10 were trying to do narrow_write_error(). I'll have a patch for that shortly.
brassow
Quoted 6 lines · NeilBrown
Quoted 5 lines
On a different topic, I've noticed the following commits in 'for-next':
90584fc MD: Prevent sysfs operations on uninitialized kobjects
e3620a3 MD RAID5: Avoid accessing gendisk or queue structs when not available
but these are not in 3.9.0-rc3. They should make their way into 3.9.0 as well as 3.8.0. (They apply cleanly to the 3.8 kernel, but I hadn't bothered to notify 'stable' - only mention the regression was introduced in 3.8-rc1.)
Quoted 7 lines
They are due to be sent to Linus today.
The second one is tagged for -stable and will go to 3.8.x (I think 3.7.x is closed).
The first isn't - my quick examination suggested that the current code is safe but not ideal. If you think it is appropriate for -stable (i.e. fixes a real bug that could hurt users) let me know why and I'll make sure it goes to -stable.
Yes, the second is the important one. We don't need the first backported to stable.
Thanks!
brassow