virtio-blk: add SGI_IO passthru support
PATCH 1/1 virtio-blk: add SGI_IO passthru support
Add support for SG_IO passthru (packet commands) to the virtio-blk backend. Conceptually based on an older patch from Hannes Reinecke but largely rewritten to match the code structure and layering in virtio-blk.
Note that currently we issue the hose SG_IO synchronously. We could easily switch to async I/O, but that would required either bloating the VirtIOBlockReq by the size of struct sg_io_hdr or an additional memory allocation for each SG_IO request.
hw/virtio-blk.h +12 -0
@@ -28,6 +28,9 @@#define VIRTIO_BLK_F_SIZE_MAX 1/* Indicates maximum segment size */ #define VIRTIO_BLK_F_SEG_MAX 2/* Indicates maximum # of segments */ #define VIRTIO_BLK_F_GEOMETRY 4/* Indicates support of legacy geometry */ +#define VIRTIO_BLK_F_RO 5/* Disk is read-only */ +#define VIRTIO_BLK_F_BLK_SIZE 6/* Block size of disk is available*/ +#define VIRTIO_BLK_F_SCSI 7/* Supports scsi command passthru */ struct virtio_blk_config {
@@ -70,6 +73,15 @@ struct virtio_blk_inhdr unsigned char status; }; +/* SCSI pass-through header */ +struct virtio_scsi_inhdr +{ +uint32_t errors; +uint32_t data_len; +uint32_t sense_len; +uint32_t residual; +}; +void *virtio_blk_init (PCIBus *bus,BlockDriverState *bs);#endif
hw/virtio-blk.c +116 -7
153 lines not on this page · raw message
PATCH 1/1 virtio-blk: add SGI_IO passthru support
[had the qemu list address wrong the first time, reply to this message,
not the previous if you were on Cc]
Add support for SG_IO passthru (packet commands) to the virtio-blk backend. Conceptually based on an older patch from Hannes Reinecke but largely rewritten to match the code structure and layering in virtio-blk.
Note that currently we issue the hose SG_IO synchronously. We could easily switch to async I/O, but that would required either bloating the VirtIOBlockReq by the size of struct sg_io_hdr or an additional memory allocation for each SG_IO request.
hw/virtio-blk.h +12 -0
@@ -28,6 +28,9 @@#define VIRTIO_BLK_F_SIZE_MAX 1/* Indicates maximum segment size */ #define VIRTIO_BLK_F_SEG_MAX 2/* Indicates maximum # of segments */ #define VIRTIO_BLK_F_GEOMETRY 4/* Indicates support of legacy geometry */ +#define VIRTIO_BLK_F_RO 5/* Disk is read-only */ +#define VIRTIO_BLK_F_BLK_SIZE 6/* Block size of disk is available*/ +#define VIRTIO_BLK_F_SCSI 7/* Supports scsi command passthru */ struct virtio_blk_config {
@@ -70,6 +73,15 @@ struct virtio_blk_inhdr unsigned char status; }; +/* SCSI pass-through header */ +struct virtio_scsi_inhdr +{ +uint32_t errors; +uint32_t data_len; +uint32_t sense_len; +uint32_t residual; +}; +void *virtio_blk_init (PCIBus *bus,BlockDriverState *bs);#endif
hw/virtio-blk.c +116 -7
153 lines not on this page · raw message
Quoted 9 lines · Christoph Hellwig
[had the qemu list address wrong the first time, reply to this message,
not the previous if you were on Cc]Add support for
SG_IOpassthru (packet commands) to the virtio-blk backend. Conceptually based on an older patch from Hannes Reinecke but largely rewritten to match the code structure and layering in virtio-blk.
Quoted 5 lines
Note that currently we issue the hose
SG_IOsynchronously. We could easily switch to async I/O, but that would required either bloating the VirtIOBlockReq by the size of structsg_io_hdror an additional memory allocation for eachSG_IOrequest.
I think that's worthwhile. The extra bloat is trivial (especially as the number of inflight virtio requests is tightly bounded), and stalling the vcpu for requests is a pain.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
I think that's worthwhile. The extra bloat is trivial (especially as the number of inflight virtio requests is tightly bounded), and stalling the vcpu for requests is a pain.
Ok, new patch will follow ASAP.
Quoted 9 lines · Christoph Hellwig
[had the qemu list address wrong the first time, reply to this message,
not the previous if you were on Cc]Add support for
SG_IOpassthru (packet commands) to the virtio-blk backend. Conceptually based on an older patch from Hannes Reinecke but largely rewritten to match the code structure and layering in virtio-blk.
Quoted 8 lines
Note that currently we issue the hose
SG_IOsynchronously. We could easily switch to async I/O, but that would required either bloating the VirtIOBlockReq by the size of structsg_io_hdror an additional memory allocation for eachSG_IOrequest.Signed-off-by Christoph Hellwig <[email protected]>
So practically speaking, what can you do with this? Should we be handling some SCSI cmds internally to QEMU (like eject operations) and supporting media=cdrom in -drive for if=virtio?
On a related topic, should we switch /dev/vdX to be /dev/sdX?
Regards,
Anthony Liguori
Quoted 1 line · Anthony Liguori
So practically speaking, what can you do with this?
It's interesting if your underlying device is a real scsi disk and you want virtio for efficiency but also allow to issue scsi commands on the underlying device, e.g. for finding out more information about the underlying setup using the sg_* tools or for certain manual multipathing setups, or for upgrading firmware or..
Quoted 3 lines
Should we be
handling some SCSI cmds internally to QEMU (like eject operations) and supporting media=cdrom in -drive for if=virtio?
Not quite yet. Eventually I want to support a virtio-scsi kind of transport which would use the same virtio-blk protocol but only send scsi commands. We'd need a different driver on the Linux side for it that registers to the scsi layer. On the QEMU side it could either do pass-through to a real scsi device using SG_IO or use the existing command scsi emulator in scsi-disk.c.
Quoted 1 line
On a related topic, should we switch
/dev/vdXto be/dev/sdX?
We can't and don't want for the current virtio-blk driver. Once we implement a virtio-scsi driver in the guest disk will show up as /dev/sdX.
Quoted 10 lines · Christoph Hellwig
Quoted 2 lines
Should we be handling some SCSI cmds internally to QEMU (like eject operations) and supporting media=cdrom in -drive for if=virtio?
Not quite yet. Eventually I want to support a virtio-scsi kind of transport which would use the same virtio-blk protocol but only send scsi commands. We'd need a different driver on the Linux side for it that registers to the scsi layer. On the QEMU side it could either do pass-through to a real scsi device using
SG_IOor use the existing command scsi emulator inscsi-disk.c.
Ah, excellent. I think that's a great thing to do. So do you think virtio-scsi would deprecate virtio-blk?
I've gotten a number of requests for exposing stuff like VPD through virtio-blk. There's also a surprising amount of software out there that assumes /dev/sdX or /dev/hdX naming.
Regards,
Anthony Liguori
Am Tuesday 28 April 2009 18:37:01 schrieb Anthony Liguori:
Quoted 14 lines
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines
Should we be
handling some SCSI cmds internally to QEMU (like eject operations) and supporting media=cdrom in -drive for if=virtio?Quoted 7 lines
Not quite yet. Eventually I want to support a virtio-scsi kind of transport which would use the same virtio-blk protocol but only send scsi commands. We'd need a different driver on the Linux side for it that registers to the scsi layer. On the QEMU side it could either do pass-through to a real scsi device using
SG_IOor use the existing command scsi emulator inscsi-disk.c.
Quoted 2 lines
Ah, excellent. I think that's a great thing to do. So do you think virtio-scsi would deprecate virtio-blk?
There are lots of other block device drivers with strange characteristics.
For example dasd/xpram disks with 4k block size or other disks with strange partitioning schemes. Doing a scsi emulation on top of these beasts looks possible but far from ideal. So I guess we will need virtio_blk for non-scsi block devices.
Christian
Quoted 2 lines · Anthony Liguori
Ah, excellent. I think that's a great thing to do. So do you think virtio-scsi would deprecate virtio-blk?
I don't think so. If you have an image format or a non-scsi blockdevice underneath virtio-block avoids the encoding into SCSI CDBs and back and should be faster.
Quoted 3 lines
I've gotten a number of requests for exposing stuff like VPD through virtio-blk. There's also a surprising amount of software out there that assumes
/dev/sdXor/dev/hdXnaming.
They should be fixed for a while, we have lots of non-scsi, non-ide block devices.
Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · Anthony Liguori
Ah, excellent. I think that's a great thing to do. So do you think virtio-scsi would deprecate virtio-blk?
I don't think so. If you have an image format or a non-scsi blockdevice underneath virtio-block avoids the encoding into SCSI CDBs and back and should be faster.
Is this actually measurably faster, or just infinitesimally faster in theory?
I can maybe see that virtio-blk is slightly simpler for dumb drivers, though even then a basic scsi host is pretty straightforward and I find it hard to believe there's much real benefit.
Paul
Am Wednesday 29 April 2009 13:11:19 schrieb Paul Brook:
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Anthony Liguori
Quoted 2 lines
Ah, excellent. I think that's a great thing to do. So do you think virtio-scsi would deprecate virtio-blk?
I don't think so. If you have an image format or a non-scsi blockdevice underneath virtio-block avoids the encoding into SCSI CDBs and back and should be faster.
Is this actually measurably faster, or just infinitesimally faster in theory?
If the underlying backing is 4k block size and the emulated scsi disk has 512 byte block size, write performance can be a lot slower.
Quoted 10 lines · Christian Borntraeger
Am Wednesday 29 April 2009 13:11:19 schrieb Paul Brook:
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Anthony Liguori
Quoted 2 lines
Ah, excellent. I think that's a great thing to do. So do you think virtio-scsi would deprecate virtio-blk?
I don't think so. If you have an image format or a non-scsi blockdevice underneath virtio-block avoids the encoding into SCSI CDBs and back and should be faster.
Quoted 5 lines
Quoted 2 lines
Is this actually measurably faster, or just infinitesimally faster in theory?
If the underlying backing is 4k block size and the emulated scsi disk has 512 byte block size, write performance can be a lot slower.
Rubbish. If this is really a problem you can just use a scsi disk with 4k sectors.
Paul
Quoted 1 line · Paul Brook
Is this actually measurably faster, or just infinitesimally faster in theory?
On normal disks it's rather theoretical. On high-end SSDs and arrays the impact is noticeable, mostly due to the additional latency.
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Paul Brook
Is this actually measurably faster, or just infinitesimally faster in theory?
On normal disks it's rather theoretical. On high-end SSDs and arrays the impact is noticeable, mostly due to the additional latency.
How exactly does it introduce additional latency? A scsi command block is hardly large or complicated. Are you suggesting that a 16/32byte scsi command takes significantly longer to process than a 16byte virtio command descriptor? I'd expect any extra processing to be a small fraction of the host syscall latency, let alone the latency of the physical host adapter. It probably even fits on the same CPU cache line.
Paul
Quoted 6 lines · Paul Brook
How exactly does it introduce additional latency? A scsi command block is hardly large or complicated. Are you suggesting that a 16/32byte scsi command takes significantly longer to process than a 16byte virtio command descriptor? I'd expect any extra processing to be a small fraction of the host syscall latency, let alone the latency of the physical host adapter. It probably even fits on the same CPU cache line.
Encoding the scsi CDB is additional work but I would be surprised it it is mesurable. Just using scsi cdbs would be simple enough, the bigger issue is emulating a full blown scsi bus because then you need to do all kinds queueing decisions at target levels etc and drag in a complicated scsi stack and not just a simple block driver in the guest. And at least on current linux kernels that does introduce mesurable latency.
Now it might be possible to get that latency down to a level where we can ignore it but when doing all this additional work there always will be additional overhead.
Quoted 2 lines
Paul
---end quoted text---
Quoted 15 lines · Christoph Hellwig
Quoted 6 lines · Paul Brook
How exactly does it introduce additional latency? A scsi command block is hardly large or complicated. Are you suggesting that a 16/32byte scsi command takes significantly longer to process than a 16byte virtio command descriptor? I'd expect any extra processing to be a small fraction of the host syscall latency, let alone the latency of the physical host adapter. It probably even fits on the same CPU cache line.
Encoding the scsi CDB is additional work but I would be surprised it it is mesurable. Just using scsi cdbs would be simple enough, the bigger issue is emulating a full blown scsi bus because then you need to do all kinds queueing decisions at target levels etc and drag in a complicated scsi stack and not just a simple block driver in the guest. And at least on current linux kernels that does introduce mesurable latency.
Quoted 4 lines
Now it might be possible to get that latency down to a level where we can ignore it but when doing all this additional work there always will be additional overhead.
/me puts on SCSI target mode hat
The other obvious benefit for allowing passthrough SCSI block devices into KVM guests via virtio-blk means that at some point those SCSI block devices could be coming from a local target mode stack that is representing LVM block devices as SCSI-3 storage, or say FILEIO on top of a btrfs mount also representing SCSI-3 storage along with the typical hardware passthroughs for local (KVM Host) accessable SCSI devices.
The important part is that KVM guests using SG_IO passthrough with virtio-blk (assuming that they actually showed up as SCSI devices in the KVM guest) is that guests would be able to take advantage of existing Linux SCSI I/O fencing functionality that is used for H/A, and for cluster filesystems like GFS in RHEL, etc.
Also using scsi_dh_alua in Linux KVM guests against SCSI-3 compatible block_devices using the passthrough means you could do some interesting things with controlling bandwith and paths using asymmetric access port states from local storage into Linux KVM guest.
How much latency overhead a SCSI passthrough (with a pseudo SCSI bus) in the KVM guest would add compared to native virtio-blk would be the key metric. Now if the KVM SCSI passthrough was talking in SCSI-target mode directly to drivers/scsi (instead of going through Block -> SCSI for example) it would actually involve less overhead on the KVM host IMHO.
For the usage of using BLOCK and FILEIO devices that have SCSI-3 emulation on top (and then passing into the KVM guest) would obviously add processing overhead, but for those folks who are interested in SCSI I/O fencing in the KVM guest using existing tools, the performance overhead would be reasonable tradeoff on machines with a fast memory and point-to-point bus architecture.
--nab
Quoted 8 lines
Quoted 2 lines
Paul
---end quoted text---
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Paul Brook
How exactly does it introduce additional latency? A scsi command block is hardly large or complicated. Are you suggesting that a 16/32byte scsi command takes significantly longer to process than a 16byte virtio command descriptor? I'd expect any extra processing to be a small fraction of the host syscall latency, let alone the latency of the physical host adapter. It probably even fits on the same CPU cache line.
Quoted 6 lines
Encoding the scsi CDB is additional work but I would be surprised it it is mesurable. Just using scsi cdbs would be simple enough, the bigger issue is emulating a full blown scsi bus because then you need to do all kinds queueing decisions at target levels etc and drag in a complicated scsi stack and not just a simple block driver in the guest. And at least on current linux kernels that does introduce mesurable latency.
Only if you emulate a crufty old parallel scsi bus, and that's just silly. One of the nice things about scsi is it separates the command set from the transport layer. cf. USB mass-storage, SAS, SBP2(firewire), and probably several others I've forgotten.
Paul
Quoted 3 lines · Paul Brook
One of the nice things about scsi is it separates the command set from the transport layer. cf. USB mass-storage, SAS, SBP2(firewire), and probably several others I've forgotten.
ATAPI, SCSIoFC, SCSIoIB....
Signature · 1 line
Javier
Quoted 4 lines · Paul Brook
Only if you emulate a crufty old parallel scsi bus, and that's just silly. One of the nice things about scsi is it separates the command set from the transport layer. cf. USB mass-storage, SAS, SBP2(firewire), and probably several others I've forgotten.
It has nothing to do with an SPI bus. Everything that resembles a SAM architecture can have multiple LUs per targer, and multiple targers per initiator port, so we need all the complex queing code, and we need error handling and and and. For some really simple ones like USB mass-storage we might get away with some ad-hoc SCSI CDB generation instead of real scsi stack, but I would recommend against it.
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Paul Brook
Only if you emulate a crufty old parallel scsi bus, and that's just silly. One of the nice things about scsi is it separates the command set from the transport layer. cf. USB mass-storage, SAS, SBP2(firewire), and probably several others I've forgotten.
It has nothing to do with an SPI bus. Everything that resembles a SAM architecture can have multiple LUs per targer, and multiple targers per initiator port, so we need all the complex queing code, and we need error handling and and and.
If you're using virtio-block to connect to lots of LUNs on lots of targets (i.e. lots of block devices), don't you need similar queuing code and error handling for all that too?
-- Jamie
Quoted 3 lines · Jamie Lokier
If you're using virtio-block to connect to lots of LUNs on lots of targets (i.e. lots of block devices), don't you need similar queuing code and error handling for all that too?
Currenly there's a 1:1 relation of virtio drivers and virtio queues.
Note that I don't argue against virtio-scsi, I brought this up first. I just thing that virtio-blk is actually nicer for some use cases. (and of course we'll have to keep it for backwards compatiblity anyway)
Quoted 15 lines · Christoph Hellwig
Quoted 6 lines · Paul Brook
How exactly does it introduce additional latency? A scsi command block is hardly large or complicated. Are you suggesting that a 16/32byte scsi command takes significantly longer to process than a 16byte virtio command descriptor? I'd expect any extra processing to be a small fraction of the host syscall latency, let alone the latency of the physical host adapter. It probably even fits on the same CPU cache line.
Encoding the scsi CDB is additional work but I would be surprised it it is mesurable. Just using scsi cdbs would be simple enough, the bigger issue is emulating a full blown scsi bus because then you need to do all kinds queueing decisions at target levels etc and drag in a complicated scsi stack and not just a simple block driver in the guest. And at least on current linux kernels that does introduce mesurable latency.
Quoted 3 lines
Now it might be possible to get that latency down to a level where we can ignore it but when doing all this additional work there always will be additional overhead.
But Paul, if you're enthusiastic you could patch lguest and show how simple it is :)
Cheers,
Rusty.
Am Tuesday 28 April 2009 11:51:54 schrieb Christoph Hellwig:
Quoted 5 lines
Not quite yet. Eventually I want to support a virtio-scsi kind of transport which would use the same virtio-blk protocol but only send scsi commands. We'd need a different driver on the Linux side for it that registers to the scsi layer. On the QEMU side it could either do pass-through to a real scsi device using
SG_IOor use the existing
Yes, virtio-scsi is also something we were thinking of. The last time we discussed this idea of SCSI passthrough internally, we stumbled over error recovery. Who is responsible for the error recovery? The host, the guest, or both? Are there problems, which will trigger error recovery in the guest and host midlayer at the same time? To be honest, my scsi knowledge is very limited, so I dont know if that is a real problem.
Christian
Quoted 1 line · Christian Borntraeger
Yes, virtio-scsi is also something we were thinking of. The last time we discussed this idea of SCSI passthrough internally, we stumbled over error recovery. Who is responsible for the error recovery? The host, the guest, or both? Are there problems, which will trigger error recovery in the guest and host midlayer at the same time? To be honest, my scsi knowledge is very limited, so I dont know if that is a real problem.
I'm not that far with planning yet. The best way would be to do real error handling in the host probably.
Am Wednesday 29 April 2009 12:50:04 schrieb Christoph Hellwig:
Quoted 5 lines · Christian Borntraeger
Quoted 1 line
Yes, virtio-scsi is also something we were thinking of. The last time we discussed this idea of SCSI passthrough internally, we stumbled over error recovery. Who is responsible for the error recovery? The host, the guest, or both? Are there problems, which will trigger error recovery in the guest and host midlayer at the same time? To be honest, my scsi knowledge is very limited, so I dont know if that is a real problem.
I'm not that far with planning yet. The best way would be to do real error handling in the host probably.
Yes, that was my impression too. The tricky part is the timeout based recovery found in the scsi midlayer (e.g. scsi-error.c). In case of a timeout the guest and the host will start recovery at the same time.
Christian
PATCH 2/2 virtio-blk: add SG_IO passthru support
In reply to Christoph HellwigAdd support for SG_IO passthru (packet commands) to the virtio-blk backend. Conceptually based on an older patch from Hannes Reinecke but largely rewritten to match the code structure and layering in virtio-blk aswell as doing asynchronous I/O.
hw/virtio-blk.h +12 -0
@@ -28,6 +28,9 @@#define VIRTIO_BLK_F_SIZE_MAX 1/* Indicates maximum segment size */ #define VIRTIO_BLK_F_SEG_MAX 2/* Indicates maximum # of segments */ #define VIRTIO_BLK_F_GEOMETRY 4/* Indicates support of legacy geometry */ +#define VIRTIO_BLK_F_RO 5/* Disk is read-only */ +#define VIRTIO_BLK_F_BLK_SIZE 6/* Block size of disk is available*/ +#define VIRTIO_BLK_F_SCSI 7/* Supports scsi command passthru */ struct virtio_blk_config {
@@ -70,6 +73,15 @@ struct virtio_blk_inhdr unsigned char status; }; +/* SCSI pass-through header */ +struct virtio_scsi_inhdr +{ +uint32_t errors; +uint32_t data_len; +uint32_t sense_len; +uint32_t residual; +}; +void *virtio_blk_init (PCIBus *bus,BlockDriverState *bs);#endif
hw/virtio-blk.c +117 -7
154 lines not on this page · raw message