: gpt_write_disklabel function robustness to sudden power off
Hello everyone,
I have a question regarding the fdisk library (libfdisk) provided in the 2.26 version of util-linux. I use it to create a MBR/GPT partition scheme on a eMMC memory. I also use the partition renaming mechanism to switch from a normal boot to an update boot (the bootloader compares partitions names to choose the one to boot on).
I was wondering if the gpt_write_disklabel function was robust to sudden power-off. In the source code, the writing procedure is as follow (UEFI requires writing in this specific order):
1) backup partition tables
2) backup GPT header
3) primary partition tables
4) primary GPT header
5) protective MBR
and uses the standard linux write function with a file descriptor. Is the writing order guaranty as operation is not synchronous ? I know that the linux io scheduler can "optimize" writing operations order. This can introduce an issue if only the primary and backup headers are written but not the partition tables.
Thank you,
Signature · 9 lines
Ronan CHAUVIN Embedded Software Engineer ASIC team -------------------------------- Parrot 174, quai de Jemmapes 75010 Paris France -------------------------------- www.parrot.com
Karel Zak
maintainerQuoted 16 lines · Ronan CHAUVIN
I have a question regarding the fdisk library (libfdisk) provided in the 2.26 version of util-linux. I use it to create a MBR/GPT partition scheme on a eMMC memory. I also use the partition renaming mechanism to switch from a normal boot to an update boot (the bootloader compares partitions names to choose the one to boot on).
I was wondering if the
gpt_write_disklabelfunction was robust to sudden power-off. In the source code, the writing procedure is as follow (UEFI requires writing in this specific order):1) backup partition tables
2) backup GPT header
3) primary partition tables
4) primary GPT header
5) protective MBR
Quoted 5 lines
and uses the standard linux write function with a file descriptor. Is the writing order guaranty as operation is not synchronous ? I know that the linux io scheduler can "optimize" writing operations order. This can introduce an issue if only the primary and backup headers are written but not the partition tables.
The order suggested by UEFI is there because GPT header contains CRC of the array with partitions and the header is validated by another top-level CRC. If you read things in reverse order (PMBR, header, partitions) and verify all the CRCs then you can be sure that all is valid.
IMHO the "right" write procedure is just holy grail... in reality we have no any guaranty (due to storage HW).
The important is to be able to detect inconsistent stuff on the device when you *read* GPT.
We can add fsync() between the steps, but I still have doubts it will improve anything. For example libparted also uses write() only.
We call fsync() before close() in libfdisk/src/context.c:fdisk_deassign_device().
Conclusion: be pessimistic and verify all you read from disk and be optimistic when you write to the disk, and when when someone is talking about write guaranty and run far away. That's all the story.
Karel
Signature · 2 lines
Karel Zak <[email protected]> http://karelzak.blogspot.com
Quoted 3 lines · Karel Zak
Conclusion: be pessimistic and verify all you read from disk and be optimistic when you write to the disk, and when when someone is talking about write guaranty and run far away. That's all the story.
The whole GPT is what, 16kiB or so? On most storage, you could force data to persistent storage with a granularity of 4kiB, with fdatasync(2) (assuming that works on block devices, not just files).
But some SSDs lie, and will claim that data is flushed to persistent storage when it isn't. (According to one of Marc Merlin's BTRFS talks).
So I'd agree with Karel that the current method is probably
ideal. write() everything, then fsync() so it all hits the disk in one multi-sector write op. Not necessarily atomic, but probably.
If we think the backup partition table / GPT header is useful, write(backup); fsync();
sleep(1sec);
write(primary); fsync();
is potentially worthwhile. On an SSD, there's the mapping metadata separate from the actual data, and the write block size might be 8kiB on some current disks. (This is why I'm thinking that the 1sec pause between writing the backup and primary would give a chance for whatever write-back caching layers to actually flush for real.)
I don't know how likely that is to help on any real storage setup;
I'm really just making that up. I also don't know whether the backup and primary are in separate 4kiB or 8kiB data blocks. Even if not, it could still be useful to always be writing blocks where one of the two copies written matches what's already there, so there's a valid table whether the old or new version is there when you try to read it back.
So I think there's potentially a tiny benefit to a fsync();sleep(), but I'd wait for confirmation from a storage expert before implementing it. The current method probably just sends one write op to the hardware for the whole GPT, which is nice.
Signature · 6 lines
#define X(x,y) x##y Peter Cordes ; e-mail: X(peter@cor , des.ca) "The gods confound the man who first found out how to distinguish the hours! Confound him, too, who in this place set up a sundial, to cut and hack my day so wretchedly into small pieces!" -- Plautus, 200 BC
Quoted 2 lines · Ronan CHAUVIN
I was wondering if the
gpt_write_disklabelfunction was robust to sudden power-off.
gpt_write_disklabel can only be "robust to sudden power-off" if there is some expectation that the partition tables, etc. can be in some sort of "consistent" state if power is lost. But it seems to me that there is no such consistent state -- what condition would you want that information to be in? The old information is irretrevably lost during the operation; the only problem that can be caused by sudden power-off is that you have to enter the new information into fdisk again before the disk has a valid structure.
Dale
Thank you for your answer.
If we are not resizing/adding but just renaming a partition, we want to be sure that at least the primary or the backup GPT header/GPT partition array will not be corrupted in the case of a sudden power-off. If the write order is not guaranty, then the primary and backup GPT headers can be written to the emmc without the corresponding partition array and the system will not be consistent.
For example, if we have this effective write operation order on the disk:
1) backup GPT header
2) primary GPT header
--> power-off <--
3) primary partition tables
4) backup partition tables
5) protective MBR
then, CRC of partition array present in both GPT headers will be incorrect.
Quoted 4 lines · Dale R. Worley
But it seems to me that there is
no such consistent state -- what condition would you want that information to be in? The old information is irretrevably lost during the operation;
Signature · 9 lines
Ronan CHAUVIN Embedded Software Engineer ASIC team -------------------------------- Parrot 174, quai de Jemmapes 75010 Paris France -------------------------------- www.parrot.com
Thank you for your answer.
Quoted 7 lines · Peter Cordes
Quoted 3 lines · Karel Zak
Conclusion: be pessimistic and verify all you read from disk and be optimistic when you write to the disk, and when when someone is talking about write guaranty and run far away. That's all the story.
The whole GPT is what, 16kiB or so? On most storage, you could force data to persistent storage with a granularity of 4kiB, with fdatasync(2) (assuming that works on block devices, not just files).
The whole GPT is 16kiB (MBR+GPT header+partition array). There is two GPT systems, one at the beginning and another one at the end. The bootloader verifies the integrity of the header and the partition array with a CRC32.
Quoted 9 lines
write()everything, thenfsync()so it all hits the disk inBut some SSDs lie, and will claim that data is flushed to persistent storage when it isn't. (According to one of Marc Merlin's BTRFS talks).
So I'd agree with Karel that the current method is probably
ideal.write()everything, thenfsync()so it all hits the disk in one multi-sector write op. Not necessarily atomic, but probably.
As the block will not be consecutive (primary and backup), the operation cannot be done in one write operation....
Quoted 17 lines
If we think the backup partition table / GPT header is useful, write(backup);
fsync();
sleep(1sec);
write(primary);fsync();
is potentially worthwhile. On an SSD, there's the mapping metadata separate from the actual data, and the write block size might be 8kiB on some current disks. (This is why I'm thinking that the 1sec pause between writing the backup and primary would give a chance for whatever write-back caching layers to actually flush for real.)I don't know how likely that is to help on any real storage setup;
I'm really just making that up. I also don't know whether the backup and primary are in separate 4kiB or 8kiB data blocks. Even if not, it could still be useful to always be writing blocks where one of the two copies written matches what's already there, so there's a valid table whether the old or new version is there when you try to read it back.
Quoted 4 lines
So I think there's potentially a tiny benefit to a fsync();sleep(), but I'd wait for confirmation from a storage expert before implementing it. The current method probably just sends one write op to the hardware for the whole GPT, which is nice.
I agree that we should wait confirmation of a storage expert but the fsync() and sleep() combination should guaranty the operation order on most hardware.
Quoted 1 line
Best regards,
Signature · 9 lines
Ronan CHAUVIN Embedded Software Engineer ASIC team -------------------------------- Parrot 174, quai de Jemmapes 75010 Paris France -------------------------------- www.parrot.com
Quoted 15 lines · Ronan CHAUVIN
Quoted 7 lines · Peter Cordes
Quoted 3 lines · Karel Zak
Conclusion: be pessimistic and verify all you read from disk and be optimistic when you write to the disk, and when when someone is talking about write guaranty and run far away. That's all the story.
The whole GPT is what, 16kiB or so? On most storage, you could force data to persistent storage with a granularity of 4kiB, with fdatasync(2) (assuming that works on block devices, not just files).
The whole GPT is 16kiB (MBR+GPT header+partition array). There is two GPT systems, one at the beginning and another one at the end. The bootloader verifies the integrity of the header and the partition array with a CRC32.
Quoted 7 lines
Quoted 5 lines
write()everything, thenfsync()so it all hits the disk inSo I'd agree with Karel that the current method is probably
ideal.write()everything, thenfsync()so it all hits the disk in one multi-sector write op. Not necessarily atomic, but probably.As the block will not be consecutive (primary and backup), the operation cannot be done in one write operation....
So at least one of the four 4kiB sectors doesn't get written at all? Because if all the sectors are getting written, regardless of order, Linux will merge the IOs into one write request to send over the SATA (or whatever) wire. Write request merging is useful even on SSDs, so Linux does it.
Even if there is a sector that doesn't get written, it's probably
still academic. Sending a request in a single write OP doesn't make it atomic. On a magnetic disk, the data will still probably all hit the platter on the same rotation, just by powering down the write head as it flies over the sector you aren't writing, so the window for a power failure to cause a problem is quite small. I'm sure SSDs are far more complicated.
Quoted 3 lines
I agree that we should wait confirmation of a storage expert but the
fsync()andsleep()combination should guaranty the operation order on most hardware.
Probably 1/10th of a second is long enough, but still short enough to
not be annoying. If you're editting the partition table of a disk that isn't idle (in which case even 1 sec might not be long enough for the write to hit disk after fdatasync()), and you don't have the system on a UPS, I think we maybe don't need to waste 0.9 seconds of everyone's time just for this hypothetical user.
Signature · 6 lines
#define X(x,y) x##y Peter Cordes ; e-mail: X(peter@cor , des.ca) "The gods confound the man who first found out how to distinguish the hours! Confound him, too, who in this place set up a sundial, to cut and hack my day so wretchedly into small pieces!" -- Plautus, 200 BC
Quoted 26 lines · Peter Cordes
Quoted 12 lines · Ronan CHAUVIN
Quoted 7 lines · Peter Cordes
Quoted 3 lines · Karel Zak
Conclusion: be pessimistic and verify all you read from disk and be optimistic when you write to the disk, and when when someone is talking about write guaranty and run far away. That's all the story.
The whole GPT is what, 16kiB or so? On most storage, you could force data to persistent storage with a granularity of 4kiB, with fdatasync(2) (assuming that works on block devices, not just files).
The whole GPT is 16kiB (MBR+GPT header+partition array). There is two GPT systems, one at the beginning and another one at the end. The bootloader verifies the integrity of the header and the partition array with a CRC32.
Quoted 7 lines
Quoted 5 lines
write()everything, thenfsync()so it all hits the disk inSo I'd agree with Karel that the current method is probably
ideal.write()everything, thenfsync()so it all hits the disk in one multi-sector write op. Not necessarily atomic, but probably.As the block will not be consecutive (primary and backup), the operation cannot be done in one write operation....
So at least one of the four 4kiB sectors doesn't get written at all? Because if all the sectors are getting written, regardless of order, Linux will merge the IOs into one write request to send over the SATA (or whatever) wire. Write request merging is useful even on SSDs, so Linux does it.
Quoted 7 lines
Even if there is a sector that doesn't get written, it's probably
still academic. Sending a request in a single write OP doesn't make it atomic. On a magnetic disk, the data will still probably all hit the platter on the same rotation, just by powering down the write head as it flies over the sector you aren't writing, so the window for a power failure to cause a problem is quite small. I'm sure SSDs are far more complicated.
The guaranty of the write OP clearly depends of the hardware... The primary/backup mechanism and CRC checks are implemented to detect these hardware failures.
Quoted 11 lines
Quoted 3 lines
I agree that we should wait confirmation of a storage expert but the
fsync()andsleep()combination should guaranty the operation order on most hardware.Probably 1/10th of a second is long enough, but still short enough to
not be annoying. If you're editting the partition table of a disk that isn't idle (in which case even 1 sec might not be long enough for the write to hit disk afterfdatasync()), and you don't have the system on a UPS, I think we maybe don't need to waste 0.9 seconds of everyone's time just for this hypothetical user.
I agree that we don't need to waste 1 second of everyone's time. Nevertheless, only a fsync() between the write operation of the backup and primary GTP systems will give more chances that data are directly written to the disk (the disk cache will be flushed).
Signature · 9 lines
Ronan CHAUVIN Embedded Software Engineer ASIC team -------------------------------- Parrot 174, quai de Jemmapes 75010 Paris France -------------------------------- www.parrot.com