lvm2 *TEMPORARY* PV failure - what happens?
I've been intrigued by the discussion of what happens when a PV fails, and have begun to wonder what would happen in the case of a transient failure of a PV.
The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single VG, and then carving LVs out of that. We plan on using the CLVM tools to fit into a clustered environment.
The arrays themselves are robust (RAID 5/6, redundant power supplies, etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.
But there is always the possiblity that we could lose the CPU, memory, bus, etc. in the iSCSI controller portion of the array, which will cause downtime, but no true loss of data. Or someone may hit the wrong power switch and just reboot the thing, taking it offline for a short time. Yes, that someone would probably be me. Shame on me.
The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will be intact when it comes back up. I suppose the analagous situation would be using LVM on a group of hot swap drives and pulling one of the disks, waiting a while, and then re-inserting it.
Can someone please walk me through the resulting steps that would happen within LVM2 (or a GFS filesystem on top of that LV) in this situation?
Thanks,
-Ty!
Signature · 6 lines
-===========================- Ty! Boyack NREL Unix Network Manager [email protected] (970) 491-1186 -===========================-
my 2c. fix me if i am wrong
either activate the VG partially, and then all LVs on other PVs are still accessible. I remember these LVs will only have RO access. Though I have no idea why.
use dm-zero to generate a fake PVs and add to VG, then allow VG to activate and access those LV. But i do not know if you access a LV that is partially or fully on this PV, what will happen.
Ming
Quoted 33 lines · Ty! Boyack
I've been intrigued by the discussion of what happens when a PV fails, and have begun to wonder what would happen in the case of a transient failure of a PV.
The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single VG, and then carving LVs out of that. We plan on using the CLVM tools to fit into a clustered environment.
The arrays themselves are robust (RAID 5/6, redundant power supplies, etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.
But there is always the possiblity that we could lose the CPU, memory, bus, etc. in the iSCSI controller portion of the array, which will cause downtime, but no true loss of data. Or someone may hit the wrong power switch and just reboot the thing, taking it offline for a short time. Yes, that someone would probably be me. Shame on me.
The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will be intact when it comes back up. I suppose the analagous situation would be using LVM on a group of hot swap drives and pulling one of the disks, waiting a while, and then re-inserting it.
Can someone please walk me through the resulting steps that would happen within LVM2 (or a GFS filesystem on top of that LV) in this situation?
Thanks,
-Ty!
It is simple to play with this type of scenario by doing:
echo offline > /sys/block/<sd dev>/device/state
and later
echo running > /sys/block/<sd dev>/device/state
I know this doesn't answer your question directly.
brassow
Quoted 54 lines · Ming Zhang
my 2c. fix me if i am wrong
either activate the VG partially, and then all LVs on other PVs are still accessible. I remember these LVs will only have RO access. Though I have no idea why.
use dm-zero to generate a fake PVs and add to VG, then allow VG to activate and access those LV. But i do not know if you access a LV that is partially or fully on this PV, what will happen.
Ming
Quoted 26 lines · Ty! Boyack
I've been intrigued by the discussion of what happens when a PV fails, and have begun to wonder what would happen in the case of a transient failure of a PV.
The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single VG, and then carving LVs out of that. We plan on using the CLVM tools to fit into a clustered environment.
The arrays themselves are robust (RAID 5/6, redundant power supplies, etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.
But there is always the possiblity that we could lose the CPU, memory, bus, etc. in the iSCSI controller portion of the array, which will cause downtime, but no true loss of data. Or someone may hit the wrong power switch and just reboot the thing, taking it offline for a short time. Yes, that someone would probably be me. Shame on me.
The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will be intact when it comes back up. I suppose the analagous situation would be using LVM on a group of hot swap drives and pulling one of the disks, waiting a while, and then re-inserting it.
Quoted 7 lines
Can someone please walk me through the resulting steps that would happen within LVM2 (or a GFS filesystem on top of that LV) in this situation?
Thanks,
-Ty!
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
assume 2 scenarios
1) this PV is under use when it is disconnected temporarily. then eventually will return r/w errors to applications. but other LVs are still accessible.
2) system is off and boot up again. for this system will complain PV with UUID ... is not found. so the only way is to partially activate VG.
am i correct here?
ming
Quoted 81 lines · Jonathan E Brassow
It is simple to play with this type of scenario by doing:
echo offline > /sys/block/<sd dev>/device/state
and later
echo running > /sys/block/<sd dev>/device/state
I know this doesn't answer your question directly.
brassow
Quoted 51 lines · Ming Zhang
my 2c. fix me if i am wrong
either activate the VG partially, and then all LVs on other PVs are still accessible. I remember these LVs will only have RO access. Though I have no idea why.
use dm-zero to generate a fake PVs and add to VG, then allow VG to activate and access those LV. But i do not know if you access a LV that is partially or fully on this PV, what will happen.
Ming
Quoted 37 lines · Ty! Boyack
I've been intrigued by the discussion of what happens when a PV fails, and have begun to wonder what would happen in the case of a transient failure of a PV.
The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single
VG, and then carving LVs out of that. We plan on using the CLVM tools to fit into a clustered environment.The arrays themselves are robust (RAID 5/6, redundant power supplies, etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.
But there is always the possiblity that we could lose the CPU, memory, bus, etc. in the iSCSI controller portion of the array, which will cause
downtime, but no true loss of data. Or someone may hit the wrong power
switch and just reboot the thing, taking it offline for a short time. Yes, that someone would probably be me. Shame on me.The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will be
intact when it comes back up. I suppose the analagous situation would be using LVM on a group of hot swap drives and pulling one of the disks,
waiting a while, and then re-inserting it.Can someone please walk me through the resulting steps that would happen
within LVM2 (or a GFS filesystem on top of that LV) in this situation?Thanks,
Quoted 9 lines
Quoted 2 lines
-Ty!
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
y, sounds right. It's pretty much what I get.
brassow
Quoted 104 lines · Ming Zhang
assume 2 scenarios
1) this PV is under use when it is disconnected temporarily. then eventually will return r/w errors to applications. but other LVs are still accessible.
2) system is off and boot up again. for this system will complain PV with UUID ... is not found. so the only way is to partially activate VG.
am i correct here?
ming
Quoted 76 lines · Jonathan E Brassow
It is simple to play with this type of scenario by doing:
echo offline > /sys/block/<sd dev>/device/state
and later
echo running > /sys/block/<sd dev>/device/state
I know this doesn't answer your question directly.
brassow
Quoted 51 lines · Ming Zhang
my 2c. fix me if i am wrong
either activate the VG partially, and then all LVs on other PVs are still accessible. I remember these LVs will only have RO access. Though I have no idea why.
use dm-zero to generate a fake PVs and add to VG, then allow VG to activate and access those LV. But i do not know if you access a LV that is partially or fully on this PV, what will happen.
Ming
Quoted 37 lines · Ty! Boyack
I've been intrigued by the discussion of what happens when a PV fails, and have begun to wonder what would happen in the case of a transient failure of a PV.
The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single
VG, and then carving LVs out of that. We plan on using the CLVM tools to fit into a clustered environment.The arrays themselves are robust (RAID 5/6, redundant power supplies, etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.
But there is always the possiblity that we could lose the CPU, memory, bus, etc. in the iSCSI controller portion of the array, which will cause
downtime, but no true loss of data. Or someone may hit the wrong power
switch and just reboot the thing, taking it offline for a short time. Yes, that someone would probably be me. Shame on me.The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will be
intact when it comes back up. I suppose the analagous situation would be using LVM on a group of hot swap drives and pulling one of the disks,
waiting a while, and then re-inserting it.Can someone please walk me through the resulting steps that would happen
within LVM2 (or a GFS filesystem on top of that LV) in this situation?Thanks,
Quoted 9 lines
Quoted 2 lines
-Ty!
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/Quoted 5 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
thx.
any idea why partially activated VGs can only allow read only access to LVs on even good PVs. I think as long we do not do LVM level change like create new/resize/delete LVs, the write access to LVs data should be allowed right?
ming
Quoted 123 lines · Jonathan E Brassow
y, sounds right. It's pretty much what I get.
brassow
Quoted 116 lines · Ming Zhang
assume 2 scenarios
1) this PV is under use when it is disconnected temporarily. then eventually will return r/w errors to applications. but other LVs are still accessible.
2) system is off and boot up again. for this system will complain PV with UUID ... is not found. so the only way is to partially activate VG.
am i correct here?
ming
Quoted 87 lines · Jonathan E Brassow
It is simple to play with this type of scenario by doing:
echo offline > /sys/block/<sd dev>/device/state
and later
echo running > /sys/block/<sd dev>/device/state
I know this doesn't answer your question directly.
brassow
Quoted 62 lines · Ming Zhang
my 2c. fix me if i am wrong
either activate the VG partially, and then all LVs on other PVs are still accessible. I remember these LVs will only have RO access. Though
I have no idea why.use dm-zero to generate a fake PVs and add to VG, then allow VG to activate and access those LV. But i do not know if you access a LV that
is partially or fully on this PV, what will happen.Ming
Quoted 46 lines · Ty! Boyack
I've been intrigued by the discussion of what happens when a PV fails,
and have begun to wonder what would happen in the case of a transient
failure of a PV.The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single
VG, and then carving LVs out of that. We plan on using the CLVM tools
to fit into a clustered environment.The arrays themselves are robust (RAID 5/6, redundant power supplies,
etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.But there is always the possiblity that we could lose the CPU, memory,
bus, etc. in the iSCSI controller portion of the array, which will cause
downtime, but no true loss of data. Or someone may hit the wrong power
switch and just reboot the thing, taking it offline for a short time.
Yes, that someone would probably be me. Shame on me.The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will
be
intact when it comes back up. I suppose the analagous situation would
be using LVM on a group of hot swap drives and pulling one of the disks,
waiting a while, and then re-inserting it.Can someone please walk me through the resulting steps that would happen
within LVM2 (or a GFS filesystem on top of that LV) in this situation?Thanks,
Quoted 9 lines
Quoted 2 lines
-Ty!
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/Quoted 5 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
The first scenario you give is most likey the one to occur. I'm thinking the server is active, has active volumes which use the iSCSI array as PVs, and may or may not have applications accessing it at the time of the failure. I'm glad to hear that the other volumes should be accessable (assuming we don't stripe across the devices). It also makes sense that the user will get a r/w error or i/o error.
I'm still wondering if the disk comes back if the LV will be available again, or if LVM will mark it as failed, and assume that all blocks on it are invalid now and forevermore. Or is this a function of the filesystem? I'll be building a test case and will certainly have fun breaking it like this. I'm glad to know of the ability to test it as Jonathan pointed out that will be a simpler test bed.
Good info - thanks folks!
-Ty!
Quoted 109 lines · Ming Zhang
assume 2 scenarios
1) this PV is under use when it is disconnected temporarily. then eventually will return r/w errors to applications. but other LVs are still accessible.
2) system is off and boot up again. for this system will complain PV with UUID ... is not found. so the only way is to partially activate VG.
am i correct here?
ming
Quoted 76 lines · Jonathan E Brassow
It is simple to play with this type of scenario by doing:
echo offline > /sys/block/<sd dev>/device/state
and later
echo running > /sys/block/<sd dev>/device/state
I know this doesn't answer your question directly.
brassow
Quoted 51 lines · Ming Zhang
my 2c. fix me if i am wrong
either activate the VG partially, and then all LVs on other PVs are still accessible. I remember these LVs will only have RO access. Though I have no idea why.
use dm-zero to generate a fake PVs and add to VG, then allow VG to activate and access those LV. But i do not know if you access a LV that is partially or fully on this PV, what will happen.
Ming
Quoted 26 lines · Ty! Boyack
I've been intrigued by the discussion of what happens when a PV fails, and have begun to wonder what would happen in the case of a transient failure of a PV.
The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single VG, and then carving LVs out of that. We plan on using the CLVM tools to fit into a clustered environment.
The arrays themselves are robust (RAID 5/6, redundant power supplies, etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.
But there is always the possiblity that we could lose the CPU, memory, bus, etc. in the iSCSI controller portion of the array, which will cause downtime, but no true loss of data. Or someone may hit the wrong power switch and just reboot the thing, taking it offline for a short time. Yes, that someone would probably be me. Shame on me.
The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will be intact when it comes back up. I suppose the analagous situation would be using LVM on a group of hot swap drives and pulling one of the disks, waiting a while, and then re-inserting it.
Quoted 9 lines
Can someone please walk me through the resulting steps that would happen within LVM2 (or a GFS filesystem on top of that LV) in this situation?
Thanks,
-Ty!
Quoted 8 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/Quoted 7 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Signature · 6 lines
-===========================- Ty! Boyack NREL Unix Network Manager [email protected] (970) 491-1186 -===========================-
Quoted 6 lines · Ty! Boyack
The first scenario you give is most likey the one to occur. I'm thinking the server is active, has active volumes which use the iSCSI array as PVs, and may or may not have applications accessing it at the time of the failure. I'm glad to hear that the other volumes should be accessable (assuming we don't stripe across the devices). It also makes sense that the user will get a r/w error or i/o error.
maybe u can consider a raid5 on top of these iscsi disks, if u app are very unhappy or buggy to see these r/w errors.
Quoted 3 lines
I'm still wondering if the disk comes back if the LV will be available again, or if LVM will mark it as failed, and assume that all blocks on
it will not automatically available i think, but with that "echo ...", u might be able to see it again.
Quoted 4 lines
it are invalid now and forevermore. Or is this a function of the filesystem? I'll be building a test case and will certainly have fun breaking it like this. I'm glad to know of the ability to test it as Jonathan pointed out that will be a simpler test bed.
keep us updated. thanks.
Quoted 124 lines
Good info - thanks folks!
-Ty!
Quoted 115 lines · Ming Zhang
assume 2 scenarios
1) this PV is under use when it is disconnected temporarily. then eventually will return r/w errors to applications. but other LVs are still accessible.
2) system is off and boot up again. for this system will complain PV with UUID ... is not found. so the only way is to partially activate VG.
am i correct here?
ming
Quoted 82 lines · Jonathan E Brassow
It is simple to play with this type of scenario by doing:
echo offline > /sys/block/<sd dev>/device/state
and later
echo running > /sys/block/<sd dev>/device/state
I know this doesn't answer your question directly.
brassow
Quoted 57 lines · Ming Zhang
my 2c. fix me if i am wrong
either activate the VG partially, and then all LVs on other PVs are still accessible. I remember these LVs will only have RO access. Though I have no idea why.
use dm-zero to generate a fake PVs and add to VG, then allow VG to activate and access those LV. But i do not know if you access a LV that is partially or fully on this PV, what will happen.
Ming
Quoted 31 lines · Ty! Boyack
I've been intrigued by the discussion of what happens when a PV fails, and have begun to wonder what would happen in the case of a transient failure of a PV.
The design I'm thinking of is a SAN environment with several multi-terabyte iSCSI arrays as PVs, being grouped together into a single
VG, and then carving LVs out of that. We plan on using the CLVM tools to fit into a clustered environment.The arrays themselves are robust (RAID 5/6, redundant power supplies, etc.) and I grant that if we lose the actual array (for example, if multiple disks fail), then we are in the situation of a true and possibly total failure of the PV and loss of it's data blocks.
But there is always the possiblity that we could lose the CPU, memory, bus, etc. in the iSCSI controller portion of the array, which will cause
downtime, but no true loss of data. Or someone may hit the wrong power
switch and just reboot the thing, taking it offline for a short time. Yes, that someone would probably be me. Shame on me.The key point is that the iSCSI disk will come back in a few minutes/hours/days depending on the failure type, and all blocks will be
intact when it comes back up. I suppose the analagous situation would be using LVM on a group of hot swap drives and pulling one of the disks,
waiting a while, and then re-inserting it.Quoted 10 lines
Can someone please walk me through the resulting steps that would happen
within LVM2 (or a GFS filesystem on top of that LV) in this situation?Thanks,
-Ty!
Quoted 8 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/Quoted 7 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 6 lines · Ming Zhang
Quoted 2 lines
I'm still wondering if the disk comes back if the LV will be available again, or if LVM will mark it as failed, and assume that all blocks on
it will not automatically available i think, but with that "echo ...", u might be able to see it again.
Actually, I think it is available once the dev comes back.
brassow