I/O errors without erros from underlying device
Hi.
4.3.0 kernel, raid6 array:
md7 : active raid6 sdg[10] sdad1[9] sdac1[8] sdag1[7] sdaf1[6] sdae1[5] sdaj1[4] sdai1[3] sdah1[2] sdn1[1]
31255089152 blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
bitmap: 1/30 pages [4KB], 65536KB chunkarray had weird failure where many disks went into failed state but remove && adding these disks "fixed" it (turns out not really fixed it).
Unfortunately now some reads fail:
pread(4, 0x1483a00, 4096, 16003680464896) = -1 EIO (Input/output error)
To reproduce used xfs_io
xfs_io -d -c "pread 16003680464896 4096" /dev/md7
pread64: Input/output error
which does pread exactly as shown above.
write also fails for that area:xfs_io -d -c "pwrite 16003680464896 4096" /dev/md7
pwrite64: Input/output error
Note that nothing is written in dmesg when that happens.
I've tried various offsets and sizes of pread and at some point that was logged:
[ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
but no error from underlying devices.
List of bad blocks:
http://sprunge.us/XSWI
What can I do now?
(loosing data from that few sectors is acceptable if the rest will be readable)
Thanks,
Signature · 5 lines
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Arkadiusz> 4.3.0 kernel, raid6 array:
I think there's a bug in the 4.3.x and 4.4-rc3 and lower with block merges. I ran into these over the weekend, where v4.2.6 was stable, but anything higher would lock up and crash on me.
So first step would be to make sure you get and test v4.4-rc4.
Arkadiusz> md7 : active raid6 sdg[10] sdad1[9] sdac1[8] sdag1[7] sdaf1[6] sdae1[5] sdaj1[4] sdai1[3] sdah1[2] sdn1[1]
Arkadiusz> 31255089152 blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
Arkadiusz> bitmap: 1/30 pages [4KB], 65536KB chunk
Arkadiusz> array had weird failure where many disks went into failed state but Arkadiusz> remove && adding these disks "fixed" it (turns out not really fixed it).
Arkadiusz> Unfortunately now some reads fail:
Arkadiusz> pread(4, 0x1483a00, 4096, 16003680464896) = -1 EIO (Input/output error)
Arkadiusz> To reproduce used xfs_io
Arkadiusz> xfs_io -d -c "pread 16003680464896 4096" /dev/md7 Arkadiusz> pread64: Input/output error
Arkadiusz> which does pread exactly as shown above.
Arkadiusz> write also fails for that area:
Arkadiusz> xfs_io -d -c "pwrite 16003680464896 4096" /dev/md7 Arkadiusz> pwrite64: Input/output error
Arkadiusz> Note that nothing is written in dmesg when that happens.
Arkadiusz> I've tried various offsets and sizes of pread and at some point that was logged:
Arkadiusz> [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Arkadiusz> but no error from underlying devices.
Arkadiusz> List of bad blocks:
Arkadiusz> http://sprunge.us/XSWI
Arkadiusz> What can I do now?
Arkadiusz> (loosing data from that few sectors is acceptable if the rest will be readable)
Arkadiusz> Thanks,
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> --
Arkadiusz> To unsubscribe from this list: send the line "unsubscribe linux-raid" in Arkadiusz> the body of a message to [email protected] Arkadiusz> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · John Stoffel
Arkadiusz> 4.3.0 kernel, raid6 array:
I think there's a bug in the 4.3.x and 4.4-rc3 and lower with block merges. I ran into these over the weekend, where v4.2.6 was stable, but anything higher would lock up and crash on me.
Well, no crashes here.
Quoted 1 line
So first step would be to make sure you get and test v4.4-rc4.
Do you know which commit there?
Quoted 49 lines
Arkadiusz> md7 : active raid6 sdg[10] sdad1[9] sdac1[8] sdag1[7] sdaf1[6]
sdae1[5] sdaj1[4] sdai1[3] sdah1[2] sdn1[1] Arkadiusz> 31255089152
blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
Arkadiusz> bitmap: 1/30 pages [4KB], 65536KB chunkArkadiusz> array had weird failure where many disks went into failed state but Arkadiusz> remove && adding these disks "fixed" it (turns out not really fixed it).
Arkadiusz> Unfortunately now some reads fail:
Arkadiusz>
pread(4, 0x1483a00, 4096, 16003680464896)= -1 EIO (Input/output error)Arkadiusz> To reproduce used
xfs_io
Arkadiusz>xfs_io-d -c "pread 16003680464896 4096"/dev/md7Arkadiusz> pread64: Input/output error
Arkadiusz> which does pread exactly as shown above.Arkadiusz> write also fails for that area:
Arkadiusz>xfs_io-d -c "pwrite 16003680464896 4096"/dev/md7Arkadiusz> pwrite64: Input/output errorArkadiusz> Note that nothing is written in dmesg when that happens.
Arkadiusz> I've tried various offsets and sizes of pread and at some point that was logged: Arkadiusz> [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Arkadiusz> but no error from underlying devices.
Arkadiusz> List of bad blocks:
Arkadiusz> http://sprunge.us/XSWIArkadiusz> What can I do now?
Arkadiusz> (loosing data from that few sectors is acceptable if the rest will be readable)
Arkadiusz> Thanks,
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> --
Arkadiusz> To unsubscribe from this list: send the line "unsubscribe linux-raid" in Arkadiusz> the body of a message to [email protected]
Arkadiusz> More majordomo info at
http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arkadiusz" == Arkadiusz Miśkiewicz <[email protected]> writes:
Arkadiusz> On Monday 07 of December 2015, John Stoffel wrote:
Arkadiusz> 4.3.0 kernel, raid6 array:
Quoted 4 lines
Quoted 4 lines
I think there's a bug in the 4.3.x and 4.4-rc3 and lower with block merges. I ran into these over the weekend, where v4.2.6 was stable, but anything higher would lock up and crash on me.
Arkadiusz> Well, no crashes here.
That's good. It was hard(er) to hit when I wasn't running KVM VMs at the same time on the server, and I was running strictly RAID1 disks, so it's hard to know.
Quoted 1 line
Quoted 1 line
So first step would be to make sure you get and test v4.4-rc4.
Arkadiusz> Do you know which commit there?
Try this, from the master lkml git repository:
2873d32ff493ecbfb7d2c7f56812ab941dda42f4
Quoted 1 line
Quoted 1 line
Arkadiusz> md7 : active raid6 sdg[10] sdad1[9] sdac1[8] sdag1[7] sdaf1[6]
Quoted 2 lines
Quoted 2 lines
sdae1[5] sdaj1[4] sdai1[3] sdah1[2] sdn1[1] Arkadiusz> 31255089152
blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
Arkadiusz> bitmap: 1/30 pages [4KB], 65536KB chunk
Quoted 1 line
Quoted 1 line
Arkadiusz> array had weird failure where many disks went into failed state
Quoted 3 lines
Quoted 3 lines
but Arkadiusz> remove && adding these disks "fixed" it (turns out not really fixed it).
Arkadiusz> Unfortunately now some reads fail:
Quoted 1 line
Quoted 1 line
Arkadiusz> pread(4, 0x1483a00, 4096, 16003680464896) = -1 EIO (Input/output
Quoted 2 lines
Quoted 2 lines
error)
Arkadiusz> To reproduce used xfs_io
Arkadiusz> xfs_io -d -c "pread 16003680464896 4096" /dev/md7 Arkadiusz> pread64: Input/output error
Arkadiusz> which does pread exactly as shown above.
Quoted 1 line
Quoted 1 line
Arkadiusz> write also fails for that area:
Arkadiusz> xfs_io -d -c "pwrite 16003680464896 4096" /dev/md7 Arkadiusz> pwrite64: Input/output error
Quoted 1 line
Quoted 1 line
Arkadiusz> Note that nothing is written in dmesg when that happens.
Quoted 1 line
Quoted 1 line
Arkadiusz> I've tried various offsets and sizes of pread and at some point
Quoted 3 lines
Quoted 3 lines
that was logged: Arkadiusz> [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Arkadiusz> but no error from underlying devices.
Quoted 1 line
Quoted 1 line
Arkadiusz> List of bad blocks:
Arkadiusz> http://sprunge.us/XSWI
Quoted 1 line
Quoted 1 line
Arkadiusz> What can I do now?
Quoted 1 line
Quoted 1 line
Arkadiusz> (loosing data from that few sectors is acceptable if the rest
Quoted 2 lines
Quoted 2 lines
will be readable)
Arkadiusz> Thanks,
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> --
Arkadiusz> To unsubscribe from this list: send the line "unsubscribe
Quoted 2 lines
Quoted 2 lines
linux-raid" in Arkadiusz> the body of a message to [email protected]
Arkadiusz> More majordomo info at
Quoted 1 line
Quoted 1 line
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · John Stoffel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arkadiusz" == Arkadiusz Miśkiewicz <[email protected]> writes:
Arkadiusz> On Monday 07 of December 2015, John Stoffel wrote:
Arkadiusz> 4.3.0 kernel, raid6 array:
Quoted 3 lines
Quoted 3 lines
I think there's a bug in the 4.3.x and 4.4-rc3 and lower with block merges. I ran into these over the weekend, where v4.2.6 was stable, but anything higher would lock up and crash on me.
Arkadiusz> Well, no crashes here.
That's good. It was hard(er) to hit when I wasn't running KVM VMs at the same time on the server, and I was running strictly RAID1 disks, so it's hard to know.
Quoted 7 lines
Quoted 1 line
Quoted 1 line
So first step would be to make sure you get and test v4.4-rc4.
Arkadiusz> Do you know which commit there?
Try this, from the master lkml git repository:
2873d32ff493ecbfb7d2c7f56812ab941dda42f4
It's merge commit. Don't see any obvious patch in that merge that would help my case.
Anyway I would expect my problem to be related to badblock lists which numbers are close to dmesg error message: [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Quoted 1 line
Quoted 1 line
Quoted 1 line
But how to repair these if write() also fails and
http://www.spinics.net/lists/raid/msg49325.html suggests that write should "fix" these (by using replacement blocks I guess) ?
Quoted 60 lines
Arkadiusz> md7 : active raid6 sdg[10] sdad1[9] sdac1[8] sdag1[7] sdaf1[6]
Quoted 2 lines
Quoted 2 lines
sdae1[5] sdaj1[4] sdai1[3] sdah1[2] sdn1[1] Arkadiusz> 31255089152
blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]Arkadiusz> bitmap: 1/30 pages [4KB], 65536KB chunk
Arkadiusz> array had weird failure where many disks went into failed state
Quoted 2 lines
Quoted 2 lines
but Arkadiusz> remove && adding these disks "fixed" it (turns out not really fixed it).
Arkadiusz> Unfortunately now some reads fail:
Arkadiusz>
pread(4, 0x1483a00, 4096, 16003680464896)= -1 EIO (Input/outputQuoted 1 line
Quoted 1 line
error)
Arkadiusz> To reproduce used
xfs_io
Arkadiusz>xfs_io-d -c "pread 16003680464896 4096"/dev/md7Arkadiusz> pread64: Input/output error
Arkadiusz> which does pread exactly as shown above.Arkadiusz> write also fails for that area:
Arkadiusz>xfs_io-d -c "pwrite 16003680464896 4096"/dev/md7Arkadiusz> pwrite64: Input/output errorArkadiusz> Note that nothing is written in dmesg when that happens.
Arkadiusz> I've tried various offsets and sizes of pread and at some point
Quoted 2 lines
Quoted 2 lines
that was logged: Arkadiusz> [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Arkadiusz> but no error from underlying devices.
Arkadiusz> List of bad blocks:
Arkadiusz> http://sprunge.us/XSWIArkadiusz> What can I do now?
Arkadiusz> (loosing data from that few sectors is acceptable if the rest
Quoted 1 line
Quoted 1 line
will be readable)
Arkadiusz> Thanks,
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> --
Arkadiusz> To unsubscribe from this list: send the line "unsubscribeQuoted 2 lines
Quoted 2 lines
linux-raid" in Arkadiusz> the body of a message to [email protected]
Arkadiusz> More majordomo info at
Quoted 1 line
Quoted 1 line
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
Signature · 8 lines
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arkadiusz" == Arkadiusz Miskiewicz <[email protected]> writes:
Quoted 1 line · John Stoffel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arkadiusz" == Arkadiusz Miśkiewicz <[email protected]> writes:
Quoted 1 line · John Stoffel
Quoted 1 line
Arkadiusz> 4.3.0 kernel, raid6 array:
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
I think there's a bug in the 4.3.x and 4.4-rc3 and lower with block merges. I ran into these over the weekend, where v4.2.6 was stable, but anything higher would lock up and crash on me.
Arkadiusz> Well, no crashes here.
Quoted 7 lines
Quoted 7 lines
That's good. It was hard(er) to hit when I wasn't running KVM VMs at the same time on the server, and I was running strictly RAID1 disks, so it's hard to know.
Quoted 1 line
Quoted 1 line
So first step would be to make sure you get and test v4.4-rc4.
Arkadiusz> Do you know which commit there?
Quoted 4 lines
Quoted 4 lines
Try this, from the master lkml git repository:
2873d32ff493ecbfb7d2c7f56812ab941dda42f4
Arkadiusz> It's merge commit. Don't see any obvious patch in that merge that would help Arkadiusz> my case.
The merge from Jens Axboe talking about blk something or other. In my case, it lead to instant lockups. In your case... hard to know. Sorry.
Arkadiusz> Anyway I would expect my problem to be related to badblock Arkadiusz> lists which numbers are close to dmesg error message: [ Arkadiusz> 848.988518] Buffer I/O error on dev md7, logical block Arkadiusz> 3907148544, async page read
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Arkadiusz> But how to repair these if write() also fails and
Arkadiusz> http://www.spinics.net/lists/raid/msg49325.html suggests that write should Arkadiusz> "fix" these (by using replacement blocks I guess) ?
Arkadiusz> md7 : active raid6 sdg[10] sdad1[9] sdac1[8] sdag1[7] sdaf1[6]
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
sdae1[5] sdaj1[4] sdai1[3] sdah1[2] sdn1[1] Arkadiusz> 31255089152
blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
Arkadiusz> bitmap: 1/30 pages [4KB], 65536KB chunk
Quoted 1 line
Quoted 1 line
Arkadiusz> array had weird failure where many disks went into failed state
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
but Arkadiusz> remove && adding these disks "fixed" it (turns out not really fixed it).
Arkadiusz> Unfortunately now some reads fail:
Quoted 1 line
Quoted 1 line
Arkadiusz> pread(4, 0x1483a00, 4096, 16003680464896) = -1 EIO (Input/output
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
error)
Arkadiusz> To reproduce used xfs_io
Arkadiusz> xfs_io -d -c "pread 16003680464896 4096" /dev/md7 Arkadiusz> pread64: Input/output error
Arkadiusz> which does pread exactly as shown above.
Quoted 1 line
Quoted 1 line
Arkadiusz> write also fails for that area:
Arkadiusz> xfs_io -d -c "pwrite 16003680464896 4096" /dev/md7 Arkadiusz> pwrite64: Input/output error
Quoted 1 line
Quoted 1 line
Arkadiusz> Note that nothing is written in dmesg when that happens.
Quoted 1 line
Quoted 1 line
Arkadiusz> I've tried various offsets and sizes of pread and at some point
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
that was logged: Arkadiusz> [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Arkadiusz> but no error from underlying devices.
Quoted 1 line
Quoted 1 line
Arkadiusz> List of bad blocks:
Arkadiusz> http://sprunge.us/XSWI
Quoted 1 line
Quoted 1 line
Arkadiusz> What can I do now?
Quoted 1 line
Quoted 1 line
Arkadiusz> (loosing data from that few sectors is acceptable if the rest
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
will be readable)
Arkadiusz> Thanks,
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> --
Arkadiusz> To unsubscribe from this list: send the line "unsubscribe
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
linux-raid" in Arkadiusz> the body of a message to [email protected]
Arkadiusz> More majordomo info at
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )
Arkadiusz> --
Arkadiusz> Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) Arkadiusz> --
Arkadiusz> To unsubscribe from this list: send the line "unsubscribe linux-raid" in Arkadiusz> the body of a message to [email protected] Arkadiusz> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Arkadiusz Miskiewicz
Anyway I would expect my problem to be related to badblock lists which numbers are close to dmesg error message: [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Quoted 1 line
Quoted 1 line
Quoted 1 line
But how to repair these if
write()also fails and
http://www.spinics.net/lists/raid/msg49325.html suggests that write should "fix" these (by using replacement blocks I guess) ?
Tried to get rid of badblock lists (well, corruption in that area is better than no access at all):
mdadm --assemble /dev/md7 --force --update=no-bbl
mdadm: Cannot remove active bbl from /dev/sdae1
mdadm: Cannot remove active bbl from /dev/sdag1
mdadm: Cannot remove active bbl from /dev/sdai1
mdadm: Cannot remove active bbl from /dev/sdn1
mdadm: Cannot remove active bbl from /dev/sdg
mdadm: Cannot remove active bbl from /dev/sdad1
mdadm: /dev/md7 has been started with 10 drives.
Is there a way to archieve that anyway?
Signature · 5 lines
Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org ) -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 24 lines · Arkadiusz Miskiewicz
Quoted 9 lines · Arkadiusz Miskiewicz
Anyway I would expect my problem to be related to badblock lists which numbers are close to dmesg error message: [ 848.988518] Buffer I/O error on dev md7, logical block 3907148544, async page read
Quoted 1 line
Quoted 1 line
Quoted 1 line
But how to repair these if
write()also fails and
http://www.spinics.net/lists/raid/msg49325.html suggests that write should "fix" these (by using replacement blocks I guess) ?Tried to get rid of badblock lists (well, corruption in that area is better than no access at all):
mdadm --assemble
/dev/md7--force --update=no-bbl
mdadm: Cannot remove active bbl from/dev/sdae1
mdadm: Cannot remove active bbl from/dev/sdag1
mdadm: Cannot remove active bbl from/dev/sdai1
mdadm: Cannot remove active bbl from/dev/sdn1
mdadm: Cannot remove active bbl from/dev/sdg
mdadm: Cannot remove active bbl from/dev/sdad1
mdadm:/dev/md7has been started with 10 drives.
Quoted 2 lines
Is there a way to archieve that anyway?
You probably have bad blocks in multiple disks in the one stripe (look in /sys/block/md7/md/dev-*/badblocks or something like that to see).
To get rid of these you would need to write to every block in the stripe. I guess I should try to find a way to make that easier.
If you like you could hack mdadm to allow you to remove the bbl even though they aren't empty.
In super1.c look for:
} else if (strcmp(update, "no-bbl") == 0) {
if (sb->feature_map & __cpu_to_le32(MD_FEATURE_BAD_BLOCKS))
pr_err("Cannot remove active bbl from %s\n",devname);
else {
sb->bblog_size = 0;
sb->bblog_shift = 0;
sb->bblog_offset = 0;
}and change it to be unconditional and also to clear MD_FEATURE_BAD_BLOCKS.
No warranty expressed or implied.
NeilBrown