virtio-blk physical block size
Subsystems (2)
PATCH v2 1/1 virtio-blk physical block size
This patch adds a physical block size attribute to virtio disks, corresponding to /sys/devices/.../physical_block_size. It is defined as the request alignment which will not trigger RMW cycles. This can be important for modern disks which use 4K physical sectors (though they still support 512 logical sectors), and for file-backed disk images (which have both the underlying filesystem block size and their own allocation granularity to consider).
Installers use this to align partitions to physical block boundaries.
Note the spec already defined blk_size as the performance rather than minimum alignment. However the driver interpreted this as the logical block size, so I updated the spec to match the driver assuming the driver predates the spec and that this is an error.
--
v2: feature bit 7 was already used by an undocumented feature; use a new bit
virtio-spec-0.8.3.lyx renamed from virtio-spec-0.8.3.lyx.orig +27 -2
@@ -4138,6 +4138,19 @@ (10) Maximum total sectors in an I/O. \end_layout +\begin_layout Description +VIRTIO_BLK_F_PHYS_BLK_SIZE(11) Physical block size of disk (writes aligned + to this will not trigger read-modify-write cycles) is in +\begin_inset Quotes erd +\end_inset + +phys_blk_size +\begin_inset Quotes erd +\end_inset + +. +\end_layout + \end_deeper \begin_layout Description Device
@@ -4214,6 +4227,11 @@ \begin_layout Plain Layout + u32 phys_blk_size; +\end_layout + +\begin_layout Plain Layout + }; \end_layout
@@ -4241,9 +4259,9 @@ \begin_layout Enumerate If the VIRTIO_BLK_F_BLK_SIZE feature is negotiated, the blk_size field can - be read to determine the optimal sector size for the driver to use. + be read to determine the sector size for the driver to use. This does not effect the units used in the protocol (always 512 bytes), - but awareness of the correct value can effect performance. + but requests must be aligned to this size. \end_layout \begin_layout Enumerate
@@ -4257,6 +4275,13 @@ No requests should be submitted which go beyond this limit. \end_layout +\begin_layout Enumerate +If the VIRTIO_BLK_F_PHYS_BLK_SIZE feature is negotiated, the phys_blk_size + field should be read to determine the hardware block size. + Smaller writes are liable to require read-modify-write cycles on behalf + of the underlying hardware, however they are still supported. +\end_layout + \begin_layout Section* Device Operation \end_layout
Quoted 8 lines · Avi Kivity
This patch adds a physical block size attribute to virtio disks, corresponding to
/sys/devices/.../physical_block_size. It is defined as the request alignment which will not trigger RMW cycles. This can be important for modern disks which use 4K physical sectors (though they still support 512 logical sectors), and for file-backed disk images (which have both the underlying filesystem block size and their own allocation granularity to consider).
Quoted 6 lines
Installers use this to align partitions to physical block boundaries.
Note the spec already defined
blk_sizeas the performance rather than minimum alignment. However the driver interpreted this as the logical block size, so I updated the spec to match the driver assuming the driver predates the spec and that this is an error.
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats BLK_SIZE as the logical and physical size (see blk_queue_logical_block_size).
I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Rusty.
Quoted 8 lines · Avi Kivity
This patch adds a physical block size attribute to virtio disks, corresponding to
/sys/devices/.../physical_block_size. It is defined as the request alignment which will not trigger RMW cycles. This can be important for modern disks which use 4K physical sectors (though they still support 512 logical sectors), and for file-backed disk images (which have both the underlying filesystem block size and their own allocation granularity to consider).
Quoted 6 lines
Installers use this to align partitions to physical block boundaries.
Note the spec already defined
blk_sizeas the performance rather than minimum alignment. However the driver interpreted this as the logical block size, so I updated the spec to match the driver assuming the driver predates the spec and that this is an error.
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats BLK_SIZE as the logical and physical size (see blk_queue_logical_block_size).
I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Rusty.
Quoted 17 lines · Rusty Russell
Quoted 8 lines · Avi Kivity
This patch adds a physical block size attribute to virtio disks, corresponding to
/sys/devices/.../physical_block_size. It is defined as the request alignment which will not trigger RMW cycles. This can be important for modern disks which use 4K physical sectors (though they still support 512 logical sectors), and for file-backed disk images (which have both the underlying filesystem block size and their own allocation granularity to consider).Quoted 7 lines
Installers use this to align partitions to physical block boundaries.
Note the spec already defined
blk_sizeas the performance rather than minimum alignment. However the driver interpreted this as the logical block size, so I updated the spec to match the driver assuming the driver predates the spec and that this is an error.
Quoted 6 lines
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats
BLK_SIZEas the logical and physical size (seeblk_queue_logical_block_size).
But we want them to be different.
Quoted 4 lines
I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Yes.
Logical block size is the minimum block size the hardware will allow. Try to write less than that, and the hardware will laugh in your face.
Physical block size is the what the logical block size would have been is software didn't suck. In theory they should be the same, but since compatibility reaons clamp the logical block size to 512, they have to differ. A disk may have a physical block size of 4096 and emulate logical block size of 512 on top of that using read-modify-write.
Or so I understand it.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 17 lines · Rusty Russell
Quoted 8 lines · Avi Kivity
This patch adds a physical block size attribute to virtio disks, corresponding to
/sys/devices/.../physical_block_size. It is defined as the request alignment which will not trigger RMW cycles. This can be important for modern disks which use 4K physical sectors (though they still support 512 logical sectors), and for file-backed disk images (which have both the underlying filesystem block size and their own allocation granularity to consider).Quoted 7 lines
Installers use this to align partitions to physical block boundaries.
Note the spec already defined
blk_sizeas the performance rather than minimum alignment. However the driver interpreted this as the logical block size, so I updated the spec to match the driver assuming the driver predates the spec and that this is an error.
Quoted 6 lines
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats
BLK_SIZEas the logical and physical size (seeblk_queue_logical_block_size).
But we want them to be different.
Quoted 4 lines
I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Yes.
Logical block size is the minimum block size the hardware will allow. Try to write less than that, and the hardware will laugh in your face.
Physical block size is the what the logical block size would have been is software didn't suck. In theory they should be the same, but since compatibility reaons clamp the logical block size to 512, they have to differ. A disk may have a physical block size of 4096 and emulate logical block size of 512 on top of that using read-modify-write.
Or so I understand it.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 7 lines · Avi Kivity
Physical block size is the what the logical block size would have been is software didn't suck. In theory they should be the same, but since compatibility reaons clamp the logical block size to 512, they have to differ. A disk may have a physical block size of 4096 and emulate logical block size of 512 on top of that using read-modify-write.
Or so I understand it.
I think that's right, but a side effect is that if you get a power failure during the read-modify-write, bytes anywhere in 4096 sector may be incorrect, so journalling (etc.) needs to use 4096 byte blocks for data integrity, even though the drive emulates smaller writes.
-- Jamie
Quoted 7 lines · Avi Kivity
Physical block size is the what the logical block size would have been is software didn't suck. In theory they should be the same, but since compatibility reaons clamp the logical block size to 512, they have to differ. A disk may have a physical block size of 4096 and emulate logical block size of 512 on top of that using read-modify-write.
Or so I understand it.
I think that's right, but a side effect is that if you get a power failure during the read-modify-write, bytes anywhere in 4096 sector may be incorrect, so journalling (etc.) needs to use 4096 byte blocks for data integrity, even though the drive emulates smaller writes.
-- Jamie
Quoted 9 lines · Rusty Russell
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats
BLK_SIZEas the logical and physical size (seeblk_queue_logical_block_size).I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Quoted 3 lines
Rusty.
---end quoted text---
Quoted 9 lines · Rusty Russell
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats
BLK_SIZEas the logical and physical size (seeblk_queue_logical_block_size).I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Quoted 3 lines
Rusty.
---end quoted text---
Quoted 28 lines · Christoph Hellwig
Quoted 9 lines · Rusty Russell
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats
BLK_SIZEas the logical and physical size (seeblk_queue_logical_block_size).I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.
Quoted 3 lines
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Yes. The current VIRTIO_BLK_F_BLK_SIZE says "please use this block size". We haven't actually specified what happens if the guest doesn't, but the spec says "must", and the Linux implementation does so AFAICT.
If we want a "soft" size, we could add that as a separate feature.
Cheers,
Rusty.
Quoted 28 lines · Christoph Hellwig
Quoted 9 lines · Rusty Russell
I thought this was what I was doing, but I have shown over and over that I have no idea about block devices.
Our current driver treats
BLK_SIZEas the logical and physical size (seeblk_queue_logical_block_size).I have no idea what "logical" vs. "physical" actually means. Anyone? Most importantly, is it some Linux-internal difference or a real I/O-visible distinction?
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.
Quoted 3 lines
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Yes. The current VIRTIO_BLK_F_BLK_SIZE says "please use this block size". We haven't actually specified what happens if the guest doesn't, but the spec says "must", and the Linux implementation does so AFAICT.
If we want a "soft" size, we could add that as a separate feature.
Cheers,
Rusty.
Quoted 22 lines · Rusty Russell
Quoted 18 lines
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.Quoted 4 lines
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Quoted 6 lines
Yes. The current
VIRTIO_BLK_F_BLK_SIZEsays "please use this block size". We haven't actually specified what happens if the guest doesn't, but the spec says "must", and the Linux implementation does so AFAICT.If we want a "soft" size, we could add that as a separate feature.
No - I agree with Christoph, there's no reason to use a 512/4096 monstrosity with virtio.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 22 lines · Rusty Russell
Quoted 18 lines
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.Quoted 4 lines
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Quoted 6 lines
Yes. The current
VIRTIO_BLK_F_BLK_SIZEsays "please use this block size". We haven't actually specified what happens if the guest doesn't, but the spec says "must", and the Linux implementation does so AFAICT.If we want a "soft" size, we could add that as a separate feature.
No - I agree with Christoph, there's no reason to use a 512/4096 monstrosity with virtio.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 27 lines · Avi Kivity
Quoted 26 lines · Rusty Russell
Quoted 18 lines
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.Quoted 4 lines
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Yes. The current
VIRTIO_BLK_F_BLK_SIZEsays "please use this block size". We haven't actually specified what happens if the guest doesn't, but the spec says "must", and the Linux implementation does so AFAICT.
Quoted 5 lines
Quoted 2 lines
If we want a "soft" size, we could add that as a separate feature.
No - I agree with Christoph, there's no reason to use a 512/4096 monstrosity with virtio.
It would be good if virtio relayed the backing device's basic topology hints, so:
- If the backing dev is a real disk with 512-byte sectors, virtio should indicate 512-byte blocks to the guest.
- If the backing dev is a real disk with 4096-byte sectors, virtio should indicate 4096-byte blocks to the guest.
With databases and filesystems, if you care about data integrity:
- If the backing dev is a real disk with 4096-byte sectors, or a file whose access is through a 4096-byte-per-page cache, virtio must indicate 4096-byte blocks otherwise guest journalling is not host-powerfail safe.
You get the idea. If there is only one parameter, it really should be at least as large as the smallest unit which may be corrupted by writes when errors occur.
-- Jamie
Quoted 27 lines · Avi Kivity
Quoted 26 lines · Rusty Russell
Quoted 18 lines
Those should be the same for any sane interface. They are for classical disk devices with larger block sizes (MO, s390 dasd) and also for the now appearing 4k sector scsi disks. But in the ide world people are concerned about dos/window legacy compatiblity so they came up with a nasty hack:
- there is a physical block size as used by the disk internally (4k initially)
- all the interfaces to the operating system still happen in the traditional 512 byte blocks to not break any existing assumptions
- to make sure modern operating systems can optimize for the larger physical sectors the disks expose this size, too.
- even worse disks can also have alignment hacks for the traditional DOS partitions tables, so that the 512 byte block zero might even have an offset into the first larger physical block. This is also exposed in the ATA identify information.Quoted 4 lines
All in all I don't think this mess is a good idea to replicate in virtio. Virtio by defintion requires virtualization aware guests, so we should just follow the SCSI way of larger real block sizes here.
Yes. The current
VIRTIO_BLK_F_BLK_SIZEsays "please use this block size". We haven't actually specified what happens if the guest doesn't, but the spec says "must", and the Linux implementation does so AFAICT.
Quoted 5 lines
Quoted 2 lines
If we want a "soft" size, we could add that as a separate feature.
No - I agree with Christoph, there's no reason to use a 512/4096 monstrosity with virtio.
It would be good if virtio relayed the backing device's basic topology hints, so:
- If the backing dev is a real disk with 512-byte sectors, virtio should indicate 512-byte blocks to the guest.
- If the backing dev is a real disk with 4096-byte sectors, virtio should indicate 4096-byte blocks to the guest.
With databases and filesystems, if you care about data integrity:
- If the backing dev is a real disk with 4096-byte sectors, or a file whose access is through a 4096-byte-per-page cache, virtio must indicate 4096-byte blocks otherwise guest journalling is not host-powerfail safe.
You get the idea. If there is only one parameter, it really should be at least as large as the smallest unit which may be corrupted by writes when errors occur.
-- Jamie
Quoted 11 lines · Jamie Lokier
It would be good if virtio relayed the backing device's basic topology hints, so:
- If the backing dev is a real disk with 512-byte sectors, virtio should indicate 512-byte blocks to the guest.
- If the backing dev is a real disk with 4096-byte sectors, virtio should indicate 4096-byte blocks to the guest.
With databases and filesystems, if you care about data integrity:
Quoted 8 lines
- If the backing dev is a real disk with 4096-byte sectors, or a file whose access is through a 4096-byte-per-page cache, virtio must indicate 4096-byte blocks otherwise guest journalling is not host-powerfail safe.
You get the idea. If there is only one parameter, it really should be at least as large as the smallest unit which may be corrupted by writes when errors occur.
It's not that easy. IDE only supports larger sectors sizes with the physical sector size attribute, not native larger sectors. While scsi does support it it's untypical and I would not expect the guests to not always get it right. So the best is to use the transport native way to express that we have larger sectors and expect the guest to do the right thing.
I've done some work on the autodetection of the larger sector sizes a while ago. But now that people brought up migration I wonder if that makes sense - if we migrate from a 512 byte sector size disk to a 4096 byte sector size disk we can't simply change guest visible attributes.
Maybe we should pick one on image creation and then stick to it. For an image format we could write down this information in the image, but for a raw images that's impossible.
Quoted 11 lines · Jamie Lokier
It would be good if virtio relayed the backing device's basic topology hints, so:
- If the backing dev is a real disk with 512-byte sectors, virtio should indicate 512-byte blocks to the guest.
- If the backing dev is a real disk with 4096-byte sectors, virtio should indicate 4096-byte blocks to the guest.
With databases and filesystems, if you care about data integrity:
Quoted 8 lines
- If the backing dev is a real disk with 4096-byte sectors, or a file whose access is through a 4096-byte-per-page cache, virtio must indicate 4096-byte blocks otherwise guest journalling is not host-powerfail safe.
You get the idea. If there is only one parameter, it really should be at least as large as the smallest unit which may be corrupted by writes when errors occur.
It's not that easy. IDE only supports larger sectors sizes with the physical sector size attribute, not native larger sectors. While scsi does support it it's untypical and I would not expect the guests to not always get it right. So the best is to use the transport native way to express that we have larger sectors and expect the guest to do the right thing.
I've done some work on the autodetection of the larger sector sizes a while ago. But now that people brought up migration I wonder if that makes sense - if we migrate from a 512 byte sector size disk to a 4096 byte sector size disk we can't simply change guest visible attributes.
Maybe we should pick one on image creation and then stick to it. For an image format we could write down this information in the image, but for a raw images that's impossible.
Quoted 4 lines · Christoph Hellwig
Maybe we should pick one on image creation and then stick to it. For an image format we could write down this information in the image, but for a raw images that's impossible.
The management system should remember it (like it remembers which images belong to which guest, and how to expose them).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Christoph Hellwig
Maybe we should pick one on image creation and then stick to it. For an image format we could write down this information in the image, but for a raw images that's impossible.
The management system should remember it (like it remembers which images belong to which guest, and how to expose them).
Signature · 1 line
error compiling committee.c: too many arguments to function