virtio scsi host draft specification, v3

Hi all,

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

Hi all,

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

Quoted 3 lines

Quoted 3 lines

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

OK, I'm impressed. This is very well written and it doesn't make any of the obvious mistakes wrt. virtio.

I assume you have an implementation, as well?

Quoted 3 lines

Quoted 3 lines

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

OK, I'm impressed. This is very well written and it doesn't make any of the obvious mistakes wrt. virtio.

I assume you have an implementation, as well?

Quoted 6 lines · Rusty Russell

Quoted 3 lines

Quoted 3 lines

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

OK, I'm impressed. This is very well written and it doesn't make any of the obvious mistakes wrt. virtio.

Thanks very much, and thanks to those who corrected my early mistakes.

Quoted 1 line

I assume you have an implementation, as well?

Unfortunately not; "we're working on it", which means I should start in July when I come back from vacation.

Do you prefer to wait for one before I make a patch to the LyX source? In the meanwhile, can you reserve a subsystem ID for me?

Paolo

Quoted 6 lines · Rusty Russell

Quoted 3 lines

Quoted 3 lines

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

OK, I'm impressed. This is very well written and it doesn't make any of the obvious mistakes wrt. virtio.

Thanks very much, and thanks to those who corrected my early mistakes.

Quoted 1 line

I assume you have an implementation, as well?

Unfortunately not; "we're working on it", which means I should start in July when I come back from vacation.

Do you prefer to wait for one before I make a patch to the LyX source? In the meanwhile, can you reserve a subsystem ID for me?

Paolo

Quoted 18 lines · Paolo Bonzini

Quoted 6 lines · Rusty Russell

Quoted 3 lines

Quoted 3 lines

 after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

OK, I'm impressed.  This is very well written and it doesn't make any of the obvious mistakes wrt. virtio.

Thanks very much, and thanks to those who corrected my early mistakes.

Quoted 1 line

I assume you have an implementation, as well?

Unfortunately not; "we're working on it", which means I should start in July when I come back from vacation.

Do you prefer to wait for one before I make a patch to the LyX source? In the meanwhile, can you reserve a subsystem ID for me?

Quoted 1 line

Paolo

Sure, you can have the next subsystem ID.

It's a pain to patch once it's in LyX, so let's get the implementation base on what you posted here an see how much it changes first...

Quoted 18 lines · Paolo Bonzini

Quoted 6 lines · Rusty Russell

Quoted 3 lines

Quoted 3 lines

 after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

OK, I'm impressed.  This is very well written and it doesn't make any of the obvious mistakes wrt. virtio.

Thanks very much, and thanks to those who corrected my early mistakes.

Quoted 1 line

I assume you have an implementation, as well?

Unfortunately not; "we're working on it", which means I should start in July when I come back from vacation.

Do you prefer to wait for one before I make a patch to the LyX source? In the meanwhile, can you reserve a subsystem ID for me?

Quoted 1 line

Paolo

Sure, you can have the next subsystem ID.

It's a pain to patch once it's in LyX, so let's get the implementation base on what you posted here an see how much it changes first...

Paolo, I'll switch the Linux guest LLD and QEMU virtio-scsi skeleton that I have to comply with the spec. Does this sound good or did you want to write these from scratch?

Paolo, I'll switch the Linux guest LLD and QEMU virtio-scsi skeleton that I have to comply with the spec. Does this sound good or did you want to write these from scratch?

Hi all,

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

The virtio SCSI host is the basis of an alternative storage stack for KVM. This stack would overcome several limitations of the current solution, virtio-blk:

1) scalability limitations: virtio-blk-over-PCI puts a strong upper limit on the number of devices that can be added to a guest. Common configurations have a limit of ~30 devices. While this can be worked around by implementing a PCI-to-PCI bridge, or by using multifunction virtio-blk devices, these solutions either have not been implemented yet, or introduce management restrictions. On the other hand, the SCSI architecture is well known for its scalability and virtio-scsi supports advanced feature such as multiqueueing.

2) limited flexibility: virtio-blk does not support all possible storage scenarios. For example, it does not allow SCSI passthrough or persistent reservations. In principle, virtio-scsi provides anything that the underlying SCSI target (be it physical storage, iSCSI or the in-kernel target) supports.

3) limited extensibility: over the time, many features have been added to virtio-blk. Each such change requires modifications to the virtio specification, to the guest drivers, and to the device model in the host. The virtio-scsi spec has been written to follow SAM conventions, and exposing new features to the guest will only require changes to the host's SCSI target implementation.

Comments are welcome.

Paolo

------------------------------->8 -----------------------------------

Virtio SCSI Host Device Spec
============================

The virtio SCSI host device groups together one or more simple virtual devices (ie. disk), and allows communicating to these devices using the SCSI protocol. An instance of the device represents a SCSI host with possibly many buses, targets and LUN attached.

The virtio SCSI device services two kinds of requests:

- command requests for a logical unit;

- task management functions related to a logical unit, target or command.

The device is also able to send out notifications about added and removed logical units.

v1:
First public version

v2:
Merged all virtqueues into one, removed separate TARGET fields

v3:
Added configuration information and reworked descriptor structure. Added back multiqueue on Avi's request, while still leaving TARGET fields out. Added dummy event and clarified some aspects of the event protocol. First version sent to a wider audience (linux-kernel and virtio lists).

Configuration
-------------

Subsystem Device ID
TBD

Virtqueues
0:controlq
1:eventq
2..n:request queues

Feature bits
VIRTIO_SCSI_F_INOUT (0) - Whether a single request can include both
read-only and write-only data buffers.

Device configuration layout
struct virtio_scsi_config {
u32 num_queues;
u32 event_info_size;
u32 sense_size;
u32 cdb_size;
}

num_queues is the total number of virtqueues exposed by the device. The driver is free to use only one request queue, or it can use more to achieve better performance.

event_info_size is the maximum size that the device will fill for buffers that the driver places in the eventq. The driver should always put buffers at least of this size.

sense_size is the maximum size of the sense data that the device will write. The default value is written by the device and will always be 96, but the driver can modify it.

cdb_size is the maximum size of the CBD that the driver will write. The default value is written by the device and will always be 32, but the driver can likewise modify it.

Device initialization
---------------------

The initialization routine should first of all discover the device's virtqueues.

The driver should then place at least a buffer in the eventq. Buffers returned by the device on the eventq may be referred to as "events" in the rest of the document.

The driver can immediately issue requests (for example, INQUIRY or REPORT LUNS) or task management functions (for example, I_T RESET).

Device operation: request queues
--------------------------------

The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue.

Requests have the following format:

     struct virtio_scsi_req_cmd {
         u8 lun[8];
         u64 id;
         u8 task_attr;
         u8 prio;
         u8 crn;
         char cdb[cdb_size];
         char dataout[];

         u8 sense[sense_size];
         u32 sense_len;
         u32 residual;
         u16 status_qualifier;
         u8 status;
         u8 response;
         char datain[];
     };

     /* command-specific response values */
     #define VIRTIO_SCSI_S_OK              0
     #define VIRTIO_SCSI_S_UNDERRUN        1
     #define VIRTIO_SCSI_S_ABORTED         2
     #define VIRTIO_SCSI_S_FAILURE         3

     /* task_attr */
     #define VIRTIO_SCSI_S_SIMPLE          0
     #define VIRTIO_SCSI_S_ORDERED         1
     #define VIRTIO_SCSI_S_HEAD            2
     #define VIRTIO_SCSI_S_ACA             3

     The lun field addresses a bus, target and logical unit in the SCSI
     host.  The id field is the command identifier as defined in SAM.

Task_attr, prio and CRN are defined in SAM. The prio field should always be zero, as command priority is explicitly not supported by this version of the device. task_attr defines the task attribute as in the table above, Note that all task attributes may be mapped to SIMPLE by the device. CRN is generally expected to be 0, but clients can provide it. The maximum CRN value defined by the protocol is 255, since CRN is stored in an 8-bit integer.

All of these fields are always read-only, as are the cdb and dataout field. sense and subsequent fields are always write-only.

The sense_len field indicates the number of bytes actually written to the sense buffer. The residual field indicates the residual size, calculated as data_length - number_of_transferred_bytes, for read or write operations.

The status byte is written by the device to be the SCSI status code.

The response byte is written by the device to be one of the following:

- VIRTIO_SCSI_S_OK when the request was completed and the status byte is filled with a SCSI status code (not necessarily "GOOD").

- VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring more data than is available in the data buffers.

- VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset or another task management function.

- VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular, if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT feature has not been negotiated, the request will be immediately returned with a response equal to VIRTIO_SCSI_S_FAILURE.

Hi all,

after some preliminary discussion on the QEMU mailing list, I present a draft specification for a virtio-based SCSI host (controller, HBA, you name it).

The virtio SCSI host is the basis of an alternative storage stack for KVM. This stack would overcome several limitations of the current solution, virtio-blk:

1) scalability limitations: virtio-blk-over-PCI puts a strong upper limit on the number of devices that can be added to a guest. Common configurations have a limit of ~30 devices. While this can be worked around by implementing a PCI-to-PCI bridge, or by using multifunction virtio-blk devices, these solutions either have not been implemented yet, or introduce management restrictions. On the other hand, the SCSI architecture is well known for its scalability and virtio-scsi supports advanced feature such as multiqueueing.

2) limited flexibility: virtio-blk does not support all possible storage scenarios. For example, it does not allow SCSI passthrough or persistent reservations. In principle, virtio-scsi provides anything that the underlying SCSI target (be it physical storage, iSCSI or the in-kernel target) supports.

3) limited extensibility: over the time, many features have been added to virtio-blk. Each such change requires modifications to the virtio specification, to the guest drivers, and to the device model in the host. The virtio-scsi spec has been written to follow SAM conventions, and exposing new features to the guest will only require changes to the host's SCSI target implementation.

Comments are welcome.

Paolo

------------------------------->8 -----------------------------------

Virtio SCSI Host Device Spec
============================

The virtio SCSI host device groups together one or more simple virtual devices (ie. disk), and allows communicating to these devices using the SCSI protocol. An instance of the device represents a SCSI host with possibly many buses, targets and LUN attached.

The virtio SCSI device services two kinds of requests:

- command requests for a logical unit;

- task management functions related to a logical unit, target or command.

The device is also able to send out notifications about added and removed logical units.

v1:
First public version

v2:
Merged all virtqueues into one, removed separate TARGET fields

v3:
Added configuration information and reworked descriptor structure. Added back multiqueue on Avi's request, while still leaving TARGET fields out. Added dummy event and clarified some aspects of the event protocol. First version sent to a wider audience (linux-kernel and virtio lists).

Configuration
-------------

Subsystem Device ID
TBD

Virtqueues
0:controlq
1:eventq
2..n:request queues

Feature bits
VIRTIO_SCSI_F_INOUT (0) - Whether a single request can include both
read-only and write-only data buffers.

Device configuration layout
struct virtio_scsi_config {
u32 num_queues;
u32 event_info_size;
u32 sense_size;
u32 cdb_size;
}

num_queues is the total number of virtqueues exposed by the device. The driver is free to use only one request queue, or it can use more to achieve better performance.

event_info_size is the maximum size that the device will fill for buffers that the driver places in the eventq. The driver should always put buffers at least of this size.

sense_size is the maximum size of the sense data that the device will write. The default value is written by the device and will always be 96, but the driver can modify it.

cdb_size is the maximum size of the CBD that the driver will write. The default value is written by the device and will always be 32, but the driver can likewise modify it.

Device initialization
---------------------

The initialization routine should first of all discover the device's virtqueues.

The driver should then place at least a buffer in the eventq. Buffers returned by the device on the eventq may be referred to as "events" in the rest of the document.

The driver can immediately issue requests (for example, INQUIRY or REPORT LUNS) or task management functions (for example, I_T RESET).

Device operation: request queues
--------------------------------

The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue.

Requests have the following format:

     struct virtio_scsi_req_cmd {
         u8 lun[8];
         u64 id;
         u8 task_attr;
         u8 prio;
         u8 crn;
         char cdb[cdb_size];
         char dataout[];

         u8 sense[sense_size];
         u32 sense_len;
         u32 residual;
         u16 status_qualifier;
         u8 status;
         u8 response;
         char datain[];
     };

     /* command-specific response values */
     #define VIRTIO_SCSI_S_OK              0
     #define VIRTIO_SCSI_S_UNDERRUN        1
     #define VIRTIO_SCSI_S_ABORTED         2
     #define VIRTIO_SCSI_S_FAILURE         3

     /* task_attr */
     #define VIRTIO_SCSI_S_SIMPLE          0
     #define VIRTIO_SCSI_S_ORDERED         1
     #define VIRTIO_SCSI_S_HEAD            2
     #define VIRTIO_SCSI_S_ACA             3

     The lun field addresses a bus, target and logical unit in the SCSI
     host.  The id field is the command identifier as defined in SAM.

Task_attr, prio and CRN are defined in SAM. The prio field should always be zero, as command priority is explicitly not supported by this version of the device. task_attr defines the task attribute as in the table above, Note that all task attributes may be mapped to SIMPLE by the device. CRN is generally expected to be 0, but clients can provide it. The maximum CRN value defined by the protocol is 255, since CRN is stored in an 8-bit integer.

All of these fields are always read-only, as are the cdb and dataout field. sense and subsequent fields are always write-only.

The sense_len field indicates the number of bytes actually written to the sense buffer. The residual field indicates the residual size, calculated as data_length - number_of_transferred_bytes, for read or write operations.

The status byte is written by the device to be the SCSI status code.

The response byte is written by the device to be one of the following:

- VIRTIO_SCSI_S_OK when the request was completed and the status byte is filled with a SCSI status code (not necessarily "GOOD").

- VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring more data than is available in the data buffers.

- VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset or another task management function.

- VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular, if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT feature has not been negotiated, the request will be immediately returned with a response equal to VIRTIO_SCSI_S_FAILURE.

If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Please do not rely in bus/target/lun here. These are leftovers from parallel SCSI and do not have any meaning on modern SCSI implementation (eg FC or SAS). Rephrase that to

The lun field is the Logical Unit Number as defined in SAM.

Quoted 2 lines

The status byte is written by the device to be the SCSI status code.

?? I doubt that exists. Make that:

The status byte is written by the device to be the status code as defined in SAM.

Quoted 15 lines

The response byte is written by the device to be one of the following:

- VIRTIO_SCSI_S_OK when the request was completed and the status byte
is filled with a SCSI status code (not necessarily "GOOD").

- VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring
more data than is available in the data buffers.

- VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset
or another task management function.

Quoted 8 lines

- VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular,
if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT
feature has not been negotiated, the request will be immediately
returned with a response equal to VIRTIO_SCSI_S_FAILURE.

And, of course:

VIRTIO_SCSI_S_DISCONNECT if the request could not be processed due to a communication failure (eg device was removed or could not be reached).

This specification implies a strict one-to-one mapping between host and target. IE there is no way of specifying more than one target per host.

If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Please do not rely in bus/target/lun here. These are leftovers from parallel SCSI and do not have any meaning on modern SCSI implementation (eg FC or SAS). Rephrase that to

The lun field is the Logical Unit Number as defined in SAM.

Quoted 2 lines

The status byte is written by the device to be the SCSI status code.

?? I doubt that exists. Make that:

The status byte is written by the device to be the status code as defined in SAM.

Quoted 15 lines

The response byte is written by the device to be one of the following:

- VIRTIO_SCSI_S_OK when the request was completed and the status byte
is filled with a SCSI status code (not necessarily "GOOD").

- VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring
more data than is available in the data buffers.

- VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset
or another task management function.

Quoted 8 lines

- VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular,
if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT
feature has not been negotiated, the request will be immediately
returned with a response equal to VIRTIO_SCSI_S_FAILURE.

And, of course:

VIRTIO_SCSI_S_DISCONNECT if the request could not be processed due to a communication failure (eg device was removed or could not be reached).

This specification implies a strict one-to-one mapping between host and target. IE there is no way of specifying more than one target per host.

Quoted 6 lines

Device operation: request queues
--------------------------------

The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue.

What about request ordering?
If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Quoted 6 lines

Device operation: request queues
--------------------------------

The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue.

What about request ordering?
If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Quoted 6 lines

If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Applied with s/device/guest/g.

Quoted 5 lines

Please do not rely in bus/target/lun here. These are leftovers from parallel SCSI and do not have any meaning on modern SCSI implementation (eg FC or SAS). Rephrase that to

The lun field is the Logical Unit Number as defined in SAM.

Ok.

Quoted 7 lines

Quoted 2 lines

The status byte is written by the device to be the SCSI status code.

?? I doubt that exists. Make that:

The status byte is written by the device to be the status code as defined in SAM.

Ok.

Quoted 23 lines

Quoted 15 lines

The response byte is written by the device to be one of the following:

- VIRTIO_SCSI_S_OK when the request was completed and the status byte
is filled with a SCSI status code (not necessarily "GOOD").

- VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring
more data than is available in the data buffers.

- VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset
or another task management function.

Quoted 8 lines

- VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular,
if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT
feature has not been negotiated, the request will be immediately
returned with a response equal to VIRTIO_SCSI_S_FAILURE.

Quoted 5 lines

And, of course:

VIRTIO_SCSI_S_DISCONNECT if the request could not be processed due to a communication failure (eg device was removed or could not be reached).

Ok.

Quoted 3 lines

This specification implies a strict one-to-one mapping between host and target. IE there is no way of specifying more than one target per host.

Actually no, the intention is to use hierarchical LUNs to support more than one target per host.

Quoted 6 lines

If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Applied with s/device/guest/g.

Quoted 5 lines

Please do not rely in bus/target/lun here. These are leftovers from parallel SCSI and do not have any meaning on modern SCSI implementation (eg FC or SAS). Rephrase that to

The lun field is the Logical Unit Number as defined in SAM.

Ok.

Quoted 7 lines

Quoted 2 lines

The status byte is written by the device to be the SCSI status code.

?? I doubt that exists. Make that:

The status byte is written by the device to be the status code as defined in SAM.

Ok.

Quoted 23 lines

Quoted 15 lines

The response byte is written by the device to be one of the following:

- VIRTIO_SCSI_S_OK when the request was completed and the status byte
is filled with a SCSI status code (not necessarily "GOOD").

- VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring
more data than is available in the data buffers.

- VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset
or another task management function.

Quoted 8 lines

- VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular,
if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT
feature has not been negotiated, the request will be immediately
returned with a response equal to VIRTIO_SCSI_S_FAILURE.

Quoted 5 lines

And, of course:

VIRTIO_SCSI_S_DISCONNECT if the request could not be processed due to a communication failure (eg device was removed or could not be reached).

Ok.

Quoted 3 lines

This specification implies a strict one-to-one mapping between host and target. IE there is no way of specifying more than one target per host.

Actually no, the intention is to use hierarchical LUNs to support more than one target per host.

Quoted 6 lines · Hannes Reinecke

Quoted 6 lines

Device operation: request queues
--------------------------------

The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue.

Quoted 7 lines

What about request ordering?
If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Maybe I misunderstand - how can this be the responsibility of the device if the device does not get the information about the original ordering of the requests?

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

Quoted 6 lines · Hannes Reinecke

Quoted 6 lines

Device operation: request queues
--------------------------------

The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue.

Quoted 7 lines

What about request ordering?
If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Maybe I misunderstand - how can this be the responsibility of the device if the device does not get the information about the original ordering of the requests?

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

Quoted 57 lines · Paolo Bonzini

Quoted 7 lines

If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Applied with s/device/guest/g.

Quoted 5 lines

Please do not rely in bus/target/lun here. These are leftovers from parallel SCSI and do not have any meaning on modern SCSI implementation (eg FC or SAS). Rephrase that to

The lun field is the Logical Unit Number as defined in SAM.

Ok.

Quoted 7 lines

Quoted 2 lines

     The status byte is written by the device to be the SCSI status code.

?? I doubt that exists. Make that:

The status byte is written by the device to be the status code as defined in SAM.

Ok.

Quoted 23 lines

Quoted 15 lines

     The response byte is written by the device to be one of the following:

     - VIRTIO_SCSI_S_OK when the request was completed and the status byte
       is filled with a SCSI status code (not necessarily "GOOD").

     - VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring
       more data than is available in the data buffers.

     - VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset
       or another task management function.

Quoted 8 lines

     - VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular,
       if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT
       feature has not been negotiated, the request will be immediately
       returned with a response equal to VIRTIO_SCSI_S_FAILURE.

Quoted 5 lines

And, of course:

VIRTIO_SCSI_S_DISCONNECT if the request could not be processed due to a communication failure (eg device was removed or could not be reached).

Quoted 9 lines

Ok.

Quoted 3 lines

This specification implies a strict one-to-one mapping between host and target. IE there is no way of specifying more than one target per host.

Actually no, the intention is to use hierarchical LUNs to support more than one target per host.

Can't.

Hierarchical LUNs is a target-internal representation.
The initiator (ie guest OS) should _not_ try to assume anything about the internal structure and just use the LUN as an opaque number.

Reason being that the LUN addressing is not unique, and there are several choices on how to represent a given LUN.
So the consensus here is that different LUN numbers represent different physical devices, regardless on the (internal) LUN representation. Which in turn means we cannot use the LUN number to convey anything else than a device identification relative to a target.

Cf SAM-3 paragraph 4.8:

A logical unit number is a field (see 4.9) containing 64 bits that identifies the logical unit within a SCSI target device when accessed by a SCSI target port.

IE the LUN is dependent on the target, but you cannot make assumptions on the target.

Consequently, it's in the hosts' responsibility to figure out the targets in the system. After that it invokes the 'scan' function from the SCSI midlayer.
You can't start from a LUN and try to figure out the targets ...

If you want to support more than on target per host you need some sort of enumeration/callback which allows the host to figure out
the number of available targets.
But in general the targets are referenced by the target port identifier as specified in the appropriate standard (eg FC or SAS).
Sadly, we don't have any standard to fall back on for this.

If, however, we decide to expose some details about the backend, we could be using the values from the backend directly.
EG we could be forwarding the SCSI target port identifier here (if backed by real hardware) or creating our own SAS-type identifier when backed by qemu block. Then we could just query the backend via a new command on the controlq
(eg 'list target ports') and wouldn't have to worry about any protocol specific details here.

Quoted 57 lines · Paolo Bonzini

Quoted 7 lines

If requests are placed on arbitrary queues you'll inevitably run on locking issues to ensure strict request ordering.
I would add here:

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Applied with s/device/guest/g.

Quoted 5 lines

Please do not rely in bus/target/lun here. These are leftovers from parallel SCSI and do not have any meaning on modern SCSI implementation (eg FC or SAS). Rephrase that to

The lun field is the Logical Unit Number as defined in SAM.

Ok.

Quoted 7 lines

Quoted 2 lines

     The status byte is written by the device to be the SCSI status code.

?? I doubt that exists. Make that:

The status byte is written by the device to be the status code as defined in SAM.

Ok.

Quoted 23 lines

Quoted 15 lines

     The response byte is written by the device to be one of the following:

     - VIRTIO_SCSI_S_OK when the request was completed and the status byte
       is filled with a SCSI status code (not necessarily "GOOD").

     - VIRTIO_SCSI_S_UNDERRUN if the content of the CDB requires transferring
       more data than is available in the data buffers.

     - VIRTIO_SCSI_S_ABORTED if the request was cancelled due to a reset
       or another task management function.

Quoted 8 lines

     - VIRTIO_SCSI_S_FAILURE for other host or guest error. In particular,
       if neither dataout nor datain is empty, and the VIRTIO_SCSI_F_INOUT
       feature has not been negotiated, the request will be immediately
       returned with a response equal to VIRTIO_SCSI_S_FAILURE.

Quoted 5 lines

And, of course:

VIRTIO_SCSI_S_DISCONNECT if the request could not be processed due to a communication failure (eg device was removed or could not be reached).

Quoted 9 lines

Ok.

Quoted 3 lines

This specification implies a strict one-to-one mapping between host and target. IE there is no way of specifying more than one target per host.

Actually no, the intention is to use hierarchical LUNs to support more than one target per host.

Can't.

Hierarchical LUNs is a target-internal representation.
The initiator (ie guest OS) should _not_ try to assume anything about the internal structure and just use the LUN as an opaque number.

Reason being that the LUN addressing is not unique, and there are several choices on how to represent a given LUN.
So the consensus here is that different LUN numbers represent different physical devices, regardless on the (internal) LUN representation. Which in turn means we cannot use the LUN number to convey anything else than a device identification relative to a target.

Cf SAM-3 paragraph 4.8:

A logical unit number is a field (see 4.9) containing 64 bits that identifies the logical unit within a SCSI target device when accessed by a SCSI target port.

IE the LUN is dependent on the target, but you cannot make assumptions on the target.

Consequently, it's in the hosts' responsibility to figure out the targets in the system. After that it invokes the 'scan' function from the SCSI midlayer.
You can't start from a LUN and try to figure out the targets ...

If you want to support more than on target per host you need some sort of enumeration/callback which allows the host to figure out
the number of available targets.
But in general the targets are referenced by the target port identifier as specified in the appropriate standard (eg FC or SAS).
Sadly, we don't have any standard to fall back on for this.

If, however, we decide to expose some details about the backend, we could be using the values from the backend directly.
EG we could be forwarding the SCSI target port identifier here (if backed by real hardware) or creating our own SAS-type identifier when backed by qemu block. Then we could just query the backend via a new command on the controlq
(eg 'list target ports') and wouldn't have to worry about any protocol specific details here.

Quoted 3 lines

Quoted 3 lines

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Maybe I misunderstand - how can this be the responsibility of the device if the device does not get the information about the original ordering of the requests?

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

Quoted 3 lines

Quoted 3 lines

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Maybe I misunderstand - how can this be the responsibility of the device if the device does not get the information about the original ordering of the requests?

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

If, however, we decide to expose some details about the backend, we could be using the values from the backend directly.
EG we could be forwarding the SCSI target port identifier here (if backed by real hardware) or creating our own SAS-type identifier when backed by qemu block. Then we could just query the backend via a new command on the controlq
(eg 'list target ports') and wouldn't have to worry about any protocol specific details here.

Of course, when doing so we would be lose the ability to freely remap LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

If, however, we decide to expose some details about the backend, we could be using the values from the backend directly.
EG we could be forwarding the SCSI target port identifier here (if backed by real hardware) or creating our own SAS-type identifier when backed by qemu block. Then we could just query the backend via a new command on the controlq
(eg 'list target ports') and wouldn't have to worry about any protocol specific details here.

Of course, when doing so we would be lose the ability to freely remap LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

Quoted 11 lines · Michael S. Tsirkin

Quoted 3 lines

Quoted 3 lines

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Maybe I misunderstand - how can this be the responsibility of the device if the device does not get the information about the original ordering of the requests?

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

Paolo

Quoted 11 lines · Michael S. Tsirkin

Quoted 3 lines

Quoted 3 lines

If a device uses more than one queue it is the responsibility of the device to ensure strict request ordering.

Maybe I misunderstand - how can this be the responsibility of the device if the device does not get the information about the original ordering of the requests?

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

Paolo

Quoted 8 lines · Hannes Reinecke

If, however, we decide to expose some details about the backend, we could be using the values from the backend directly.
EG we could be forwarding the SCSI target port identifier here (if backed by real hardware) or creating our own SAS-type identifier when backed by qemu block. Then we could just query the backend via a new command on the controlq
(eg 'list target ports') and wouldn't have to worry about any protocol specific details here.

Besides the controlq command, which I can certainly add, this is actually quite similar to what I had in mind (though my plan likely would not have worked because I was expecting hierarchical LUNs used uniformly).  So, "list target ports" would return a set of LUN values to which you can send REPORT LUNS, or something like that?

If I understand it correctly, it should remain possible to use a single host for both pass-through and emulated targets.

Quoted 4 lines

Of course, when doing so we would be lose the ability to freely remap LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

And you could always use the QEMU block backend with scsi-generic if you want to remap LUNs, instead of true passthrough via the kernel target.

Quoted 8 lines · Hannes Reinecke

If, however, we decide to expose some details about the backend, we could be using the values from the backend directly.
EG we could be forwarding the SCSI target port identifier here (if backed by real hardware) or creating our own SAS-type identifier when backed by qemu block. Then we could just query the backend via a new command on the controlq
(eg 'list target ports') and wouldn't have to worry about any protocol specific details here.

Besides the controlq command, which I can certainly add, this is actually quite similar to what I had in mind (though my plan likely would not have worked because I was expecting hierarchical LUNs used uniformly).  So, "list target ports" would return a set of LUN values to which you can send REPORT LUNS, or something like that?

If I understand it correctly, it should remain possible to use a single host for both pass-through and emulated targets.

Quoted 4 lines

Of course, when doing so we would be lose the ability to freely remap LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

And you could always use the QEMU block backend with scsi-generic if you want to remap LUNs, instead of true passthrough via the kernel target.

Which is exactly the problem I was referring to. When using more than one channel the request ordering _as seen by the initiator_ has to be preserved.

This is quite hard to do from a device's perspective; it might be able to process the requests _in the order_ they've arrived, but it won't be able to figure out the latency of each request, ie how it'll take the request to be delivered to the initiator.

What we need to do here is to ensure that virtio will deliver the requests in-order across all virtqueues. Not sure whether it does this already.

Which is exactly the problem I was referring to. When using more than one channel the request ordering _as seen by the initiator_ has to be preserved.

This is quite hard to do from a device's perspective; it might be able to process the requests _in the order_ they've arrived, but it won't be able to figure out the latency of each request, ie how it'll take the request to be delivered to the initiator.

What we need to do here is to ensure that virtio will deliver the requests in-order across all virtqueues. Not sure whether it does this already.

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

For example, if the driver is crazy enough to put all write requests on one queue and all barriers on another one, how is the device supposed to ensure ordering?

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

Quoted 5 lines

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

That doesn't really fit very well with the SAM model. If we want to use multiple queues for a single LUN it has to be transparent to the SCSI command stream. Then again I don't quite see the use for that anyway.

Quoted 5 lines

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

That doesn't really fit very well with the SAM model. If we want to use multiple queues for a single LUN it has to be transparent to the SCSI command stream. Then again I don't quite see the use for that anyway.

I think we're missing a level of addressing. We need the ability to talk to multiple target ports in order for "list target ports" to make sense. Right now there is one implicit target that handles all commands. That means there is one fixed I_T Nexus.

If we introduce "list target ports" we also need a way to say "This CDB is destined for target port #0". Then it is possible to enumerate target ports and address targets independently of the LUN field in the CDB.

I'm pretty sure this is also how SAS and other transports work. In their framing they include the target port.

The question is whether we really need to support multiple targets on a virtio-scsi adapter or not. If you are selectively mapping LUNs that the guest may access, then multiple targets are not necessary. If we want to do pass-through of the entire SCSI bus then we need multiple targets but I'm not sure if there are other challenges like dependencies on the transport (Fibre Channel, SAS, etc) which make it impossible to pass through bus-level access?

I think we're missing a level of addressing. We need the ability to talk to multiple target ports in order for "list target ports" to make sense. Right now there is one implicit target that handles all commands. That means there is one fixed I_T Nexus.

If we introduce "list target ports" we also need a way to say "This CDB is destined for target port #0". Then it is possible to enumerate target ports and address targets independently of the LUN field in the CDB.

I'm pretty sure this is also how SAS and other transports work. In their framing they include the target port.

The question is whether we really need to support multiple targets on a virtio-scsi adapter or not. If you are selectively mapping LUNs that the guest may access, then multiple targets are not necessary. If we want to do pass-through of the entire SCSI bus then we need multiple targets but I'm not sure if there are other challenges like dependencies on the transport (Fibre Channel, SAS, etc) which make it impossible to pass through bus-level access?

Quoted 10 lines · Stefan Hajnoczi

I think we're missing a level of addressing. We need the ability to talk to multiple target ports in order for "list target ports" to make sense. Right now there is one implicit target that handles all commands. That means there is one fixed I_T Nexus.

If we introduce "list target ports" we also need a way to say "This CDB is destined for target port #0". Then it is possible to enumerate target ports and address targets independently of the LUN field in the CDB.

Quoted 2 lines

I'm pretty sure this is also how SAS and other transports work. In their framing they include the target port.

Yes, exactly. Hierachial LUNs are a nasty fringe feature that we should avoid as much as possible, that is for everything but IBM vSCSI which is braindead enough to force them.

Quoted 7 lines

The question is whether we really need to support multiple targets on a virtio-scsi adapter or not. If you are selectively mapping LUNs that the guest may access, then multiple targets are not necessary. If we want to do pass-through of the entire SCSI bus then we need multiple targets but I'm not sure if there are other challenges like dependencies on the transport (Fibre Channel, SAS, etc) which make it impossible to pass through bus-level access?

I don't think bus-level pass through is either easily possible nor desirable. What multiple targets are useful for is allowing more virtual disks than we have virtual PCI slots. We could do this by supporting multiple LUNs, but given that many SCSI ressources are target-based doing multiple targets most likely is the more scabale and more logical variant. E.g. we could much more easily have one virtqueue per target than per LUN.

Quoted 10 lines · Stefan Hajnoczi

I think we're missing a level of addressing. We need the ability to talk to multiple target ports in order for "list target ports" to make sense. Right now there is one implicit target that handles all commands. That means there is one fixed I_T Nexus.

If we introduce "list target ports" we also need a way to say "This CDB is destined for target port #0". Then it is possible to enumerate target ports and address targets independently of the LUN field in the CDB.

Quoted 2 lines

I'm pretty sure this is also how SAS and other transports work. In their framing they include the target port.

Yes, exactly. Hierachial LUNs are a nasty fringe feature that we should avoid as much as possible, that is for everything but IBM vSCSI which is braindead enough to force them.

Quoted 7 lines

The question is whether we really need to support multiple targets on a virtio-scsi adapter or not. If you are selectively mapping LUNs that the guest may access, then multiple targets are not necessary. If we want to do pass-through of the entire SCSI bus then we need multiple targets but I'm not sure if there are other challenges like dependencies on the transport (Fibre Channel, SAS, etc) which make it impossible to pass through bus-level access?

I don't think bus-level pass through is either easily possible nor desirable. What multiple targets are useful for is allowing more virtual disks than we have virtual PCI slots. We could do this by supporting multiple LUNs, but given that many SCSI ressources are target-based doing multiple targets most likely is the more scabale and more logical variant. E.g. we could much more easily have one virtqueue per target than per LUN.

The general idea here is that we can support NPIV.
With NPIV we'll have several scsi_hosts, each of which is assigned a different set of LUNs by the array.
With virtio we need to able to react on LUN remapping on the array side, ie we need to be able to issue a 'REPORT LUNS' command and add/remove LUNs on the fly. This means we have to expose the scsi_host in some way via virtio.

This is impossible with a one-to-one mapping between targets and LUNs. The actual bus-level pass-through will be just on the SCSI layer, ie 'REPORT LUNS' should be possible. If and how we do a LUN remapping internally on the host is a totally different matter. Same goes for the transport details; I doubt we will expose all the dingy details of the various transports, but rather restrict ourselves to an abstract transport.

The general idea here is that we can support NPIV.
With NPIV we'll have several scsi_hosts, each of which is assigned a different set of LUNs by the array.
With virtio we need to able to react on LUN remapping on the array side, ie we need to be able to issue a 'REPORT LUNS' command and add/remove LUNs on the fly. This means we have to expose the scsi_host in some way via virtio.

This is impossible with a one-to-one mapping between targets and LUNs. The actual bus-level pass-through will be just on the SCSI layer, ie 'REPORT LUNS' should be possible. If and how we do a LUN remapping internally on the host is a totally different matter. Same goes for the transport details; I doubt we will expose all the dingy details of the various transports, but rather restrict ourselves to an abstract transport.

Quoted 10 lines · Christoph Hellwig

Quoted 5 lines

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

That doesn't really fit very well with the SAM model. If we want to use multiple queues for a single LUN it has to be transparent to the SCSI command stream. Then again I don't quite see the use for that anyway.

Agreed, I see a use for multiple queues (MSI-X), but not for multiple queues shared by a single LUN.

Paolo

Quoted 10 lines · Christoph Hellwig

Quoted 5 lines

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

That doesn't really fit very well with the SAM model. If we want to use multiple queues for a single LUN it has to be transparent to the SCSI command stream. Then again I don't quite see the use for that anyway.

Agreed, I see a use for multiple queues (MSI-X), but not for multiple queues shared by a single LUN.

Paolo

Quoted 12 lines · Paolo Bonzini

Quoted 10 lines · Christoph Hellwig

Quoted 5 lines

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

That doesn't really fit very well with the SAM model. If we want to use multiple queues for a single LUN it has to be transparent to the SCSI command stream. Then again I don't quite see the use for that anyway.

Quoted 2 lines

Agreed, I see a use for multiple queues (MSI-X), but not for multiple queues shared by a single LUN.

Then let's make it explicit in the spec?

Quoted 12 lines · Paolo Bonzini

Quoted 10 lines · Christoph Hellwig

Quoted 5 lines

I agree here, in fact I misread Hannes's comment as "if a driver uses more than one queue it is responsibility of the driver to ensure strict request ordering". If you send requests to different queues, you know that those requests are independent. I don't think anything else is feasible in the virtio framework.

That doesn't really fit very well with the SAM model. If we want to use multiple queues for a single LUN it has to be transparent to the SCSI command stream. Then again I don't quite see the use for that anyway.

Quoted 2 lines

Agreed, I see a use for multiple queues (MSI-X), but not for multiple queues shared by a single LUN.

Then let's make it explicit in the spec?

Quoted 5 lines

Quoted 4 lines

Of course, when doing so we would be lose the ability to freely remap LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

Quoted 2 lines

And you could always use the QEMU block backend with scsi-generic if you want to remap LUNs, instead of true passthrough via the kernel target.

IIUC the in-kernel target always does remapping. It passes through individual LUNs rather than entire targets and you pick LU Numbers to map to the backing storage (which may or may not be a SCSI pass-through device). Nicholas Bellinger can confirm whether this is correct.

Quoted 5 lines

Quoted 4 lines

Of course, when doing so we would be lose the ability to freely remap LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

Quoted 2 lines

And you could always use the QEMU block backend with scsi-generic if you want to remap LUNs, instead of true passthrough via the kernel target.

IIUC the in-kernel target always does remapping. It passes through individual LUNs rather than entire targets and you pick LU Numbers to map to the backing storage (which may or may not be a SCSI pass-through device). Nicholas Bellinger can confirm whether this is correct.

Quoted 10 lines · Stefan Hajnoczi

Quoted 1 line

Quoted 1 line

Of course, when doing so we would be lose the ability to

freely remap

Quoted 7 lines

Quoted 3 lines

LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

And you could always use the QEMU block backend with scsi-generic if you want to remap LUNs, instead of true passthrough via the kernel target.

Quoted 5 lines

IIUC the in-kernel target always does remapping. It passes through individual LUNs rather than entire targets and you pick LU Numbers to map to the backing storage (which may or may not be a SCSI pass-through device). Nicholas Bellinger can confirm whether this is correct.

But then I don't understand. If you pick LU numbers both with the in-kernel target and with QEMU, you do not need to use e.g. WWPNs with fiber channel, because we are not passing through the details of the transport protocol (one day we might have virtio-fc, but more likely not). So the LUNs you use might as well be represented by hierarchical LUNs.

Using NPIV with KVM would be done by mapping the same virtual N_Port ID in the host(s) to the same LU number in the guest. You might already do this now with virtio-blk, in fact.

Put in another way: the virtio-scsi device is itself a SCSI target, so yes, there is a single target port identifier in virtio-scsi. But this SCSI target just passes requests down to multiple real targets, and so will let you do ALUA and all that.

Quoted 10 lines · Stefan Hajnoczi

Quoted 1 line

Quoted 1 line

Of course, when doing so we would be lose the ability to

freely remap

Quoted 7 lines

Quoted 3 lines

LUNs. But then remapping LUNs doesn't gain you much imho. Plus you could always use qemu block backend here if you want to hide the details.

And you could always use the QEMU block backend with scsi-generic if you want to remap LUNs, instead of true passthrough via the kernel target.

Quoted 5 lines

IIUC the in-kernel target always does remapping. It passes through individual LUNs rather than entire targets and you pick LU Numbers to map to the backing storage (which may or may not be a SCSI pass-through device). Nicholas Bellinger can confirm whether this is correct.

But then I don't understand. If you pick LU numbers both with the in-kernel target and with QEMU, you do not need to use e.g. WWPNs with fiber channel, because we are not passing through the details of the transport protocol (one day we might have virtio-fc, but more likely not). So the LUNs you use might as well be represented by hierarchical LUNs.

Using NPIV with KVM would be done by mapping the same virtual N_Port ID in the host(s) to the same LU number in the guest. You might already do this now with virtio-blk, in fact.

Put in another way: the virtio-scsi device is itself a SCSI target, so yes, there is a single target port identifier in virtio-scsi. But this SCSI target just passes requests down to multiple real targets, and so will let you do ALUA and all that.

Actually, the kernel does _not_ do a LUN remapping.

Quoted 2 lines

Put in another way: the virtio-scsi device is itself a SCSI target,

Argl. No way. The virtio-scsi device has to map to a single LUN.

The SCSI spec itself only deals with LUNs, so anything you'll read in there obviously will only handle the interaction between the initiator (read: host) and the LUN itself. However, the actual command is send via an intermediat target, hence you'll always see the reference to the ITL (initiator-target-lun) nexus.

The SCSI spec details discovery of the individual LUNs presented by a given target, it does _NOT_ detail the discovery of the targets themselves. That is being delegated to the underlying transport

In the case of NPIV it would make sense to map the virtual SCSI host
to the backend, so that all devices presented to the virtual SCSI
host will be presented to the backend, too. However, when doing so these devices will normally be referenced by their original LUN, as these will be presented to the guest via eg 'REPORT LUNS'.

The above thread now tries to figure out if we should remap those LUN numbers or just expose them as they are. If we decide on remapping, we have to emulate _all_ commands referring explicitely to those LUN numbers (persistent reservations, anyone?).

If we don't, we would expose some hardware detail to the guest, but would save us _a lot_ of processing.

Actually, the kernel does _not_ do a LUN remapping.

Quoted 2 lines

Put in another way: the virtio-scsi device is itself a SCSI target,

Argl. No way. The virtio-scsi device has to map to a single LUN.

The SCSI spec itself only deals with LUNs, so anything you'll read in there obviously will only handle the interaction between the initiator (read: host) and the LUN itself. However, the actual command is send via an intermediat target, hence you'll always see the reference to the ITL (initiator-target-lun) nexus.

The SCSI spec details discovery of the individual LUNs presented by a given target, it does _NOT_ detail the discovery of the targets themselves. That is being delegated to the underlying transport

In the case of NPIV it would make sense to map the virtual SCSI host
to the backend, so that all devices presented to the virtual SCSI
host will be presented to the backend, too. However, when doing so these devices will normally be referenced by their original LUN, as these will be presented to the guest via eg 'REPORT LUNS'.

The above thread now tries to figure out if we should remap those LUN numbers or just expose them as they are. If we decide on remapping, we have to emulate _all_ commands referring explicitely to those LUN numbers (persistent reservations, anyone?).

If we don't, we would expose some hardware detail to the guest, but would save us _a lot_ of processing.

Quoted 1 line

Actually, the kernel does _not_ do a LUN remapping.

Not the kernel, the in-kernel target. The in-kernel target can and will map hardware LUNs (target_lun in drivers/target/*) to arbitrary LUNs (mapped_lun).

Quoted 4 lines

Quoted 2 lines

Put in another way: the virtio-scsi device is itself a SCSI target,

Argl. No way. The virtio-scsi device has to map to a single LUN.

I think we are talking about different things. By "virtio-scsi device" I meant the "virtio-scsi HBA". When I referred to a LUN as seen by the guest, I was calling it a "virtual SCSI device". So yes, we were calling things with different names. Perhaps from now on
we can call them virtio-scsi {initiator,target,LUN} and have no ambiguity? I'll also modify the spec in this sense.

Quoted 5 lines

The SCSI spec itself only deals with LUNs, so anything you'll read in there obviously will only handle the interaction between the initiator (read: host) and the LUN itself. However, the actual command is send via an intermediat target, hence you'll always see the reference to the ITL (initiator-target-lun) nexus.

Yes, this I understand.

Quoted 3 lines

The SCSI spec details discovery of the individual LUNs presented by a given target, it does _NOT_ detail the discovery of the targets themselves. That is being delegated to the underlying transport

And in fact I have this in virtio-scsi too, since virtio-scsi _is_ a transport:

When VIRTIO_SCSI_EVT_RESET_REMOVED or VIRTIO_SCSI_EVT_RESET_RESCAN is sent for LUN 0, the driver should ask the initiator to rescan the target, in order to detect the case when an entire target has appeared or disappeared.

[If the device fails] to report an event due to missing buffers, [...] the driver should poll the logical units for unit attention conditions, and/or do whatever form of bus scan is appropriate for the guest operating system.

Quoted 5 lines

In the case of NPIV it would make sense to map the virtual SCSI host
to the backend, so that all devices presented to the virtual SCSI
host will be presented to the backend, too. However, when doing so these devices will normally be referenced by their original LUN, as these will be presented to the guest via eg 'REPORT LUNS'.

Right.

Quoted 4 lines

The above thread now tries to figure out if we should remap those LUN numbers or just expose them as they are. If we decide on remapping, we have to emulate _all_ commands referring explicitely to those LUN numbers (persistent reservations, anyone?).

But it seems to me that commands referring explicitly to LUN numbers most likely have to be reimplemented anyway for virtualization. I'm thinking exactly of persistent reservations. If two guests on the same host try a persistent reservation, they should conflict with each other. If reservation commands were just passed through, they would be seen as coming from the same initiator (the HBA driver or iSCSI initiator in the host OS).

etc.

Quoted 2 lines

If we don't, we would expose some hardware detail to the guest, but would save us _a lot_ of processing.

But can we afford it? And would the architecture allow that at all?

Paolo

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 1 line

Actually, the kernel does _not_ do a LUN remapping.

Not the kernel, the in-kernel target. The in-kernel target can and will map hardware LUNs (target_lun in drivers/target/*) to arbitrary LUNs (mapped_lun).

Quoted 4 lines

Quoted 2 lines

Put in another way: the virtio-scsi device is itself a SCSI target,

Argl. No way. The virtio-scsi device has to map to a single LUN.

I think we are talking about different things. By "virtio-scsi device" I meant the "virtio-scsi HBA". When I referred to a LUN as seen by the guest, I was calling it a "virtual SCSI device". So yes, we were calling things with different names. Perhaps from now on
we can call them virtio-scsi {initiator,target,LUN} and have no ambiguity? I'll also modify the spec in this sense.

Quoted 5 lines

The SCSI spec itself only deals with LUNs, so anything you'll read in there obviously will only handle the interaction between the initiator (read: host) and the LUN itself. However, the actual command is send via an intermediat target, hence you'll always see the reference to the ITL (initiator-target-lun) nexus.

Yes, this I understand.

Quoted 3 lines

The SCSI spec details discovery of the individual LUNs presented by a given target, it does _NOT_ detail the discovery of the targets themselves. That is being delegated to the underlying transport

And in fact I have this in virtio-scsi too, since virtio-scsi _is_ a transport:

When VIRTIO_SCSI_EVT_RESET_REMOVED or VIRTIO_SCSI_EVT_RESET_RESCAN is sent for LUN 0, the driver should ask the initiator to rescan the target, in order to detect the case when an entire target has appeared or disappeared.

[If the device fails] to report an event due to missing buffers, [...] the driver should poll the logical units for unit attention conditions, and/or do whatever form of bus scan is appropriate for the guest operating system.

Quoted 5 lines

In the case of NPIV it would make sense to map the virtual SCSI host
to the backend, so that all devices presented to the virtual SCSI
host will be presented to the backend, too. However, when doing so these devices will normally be referenced by their original LUN, as these will be presented to the guest via eg 'REPORT LUNS'.

Right.

Quoted 4 lines

The above thread now tries to figure out if we should remap those LUN numbers or just expose them as they are. If we decide on remapping, we have to emulate _all_ commands referring explicitely to those LUN numbers (persistent reservations, anyone?).

But it seems to me that commands referring explicitly to LUN numbers most likely have to be reimplemented anyway for virtualization. I'm thinking exactly of persistent reservations. If two guests on the same host try a persistent reservation, they should conflict with each other. If reservation commands were just passed through, they would be seen as coming from the same initiator (the HBA driver or iSCSI initiator in the host OS).

etc.

Quoted 2 lines

If we don't, we would expose some hardware detail to the guest, but would save us _a lot_ of processing.

But can we afford it? And would the architecture allow that at all?

Paolo

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

So if I understand correctly, virtio-scsi looks like an SCSI tranport protocol,
such as iSCSI, FCP and SRP which use tcp/ip, FC and Infiniband RDMA respectively as the transfer media while virtio-scsi uses virtio, an virtual IO
channel, as the transfer media?

So if I understand correctly, virtio-scsi looks like an SCSI tranport protocol,
such as iSCSI, FCP and SRP which use tcp/ip, FC and Infiniband RDMA respectively as the transfer media while virtio-scsi uses virtio, an virtual IO
channel, as the transfer media?