pvSCSI (SCSI pass through) driver

Hi all,

I will send total five patches on following e-mails.

These patches provide functionality that guest domain can issue complete SCSI command to FC-SCSI HBA(Fibre Channel SCSI Host Bus Adapter) assigned to the guest domain.
Using this functionality, guest domain can control such as tape device for backup and so on.

We hope these patches will be merged into Xen 3.2.0.

Thanks,

-----
Jun Kamada

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

This seems to have been hanging around for ages. If there's no

otherwise I'll give it a read through and stick it in for 3.2.0.

Quoted 4 lines

These patches provide functionality that guest domain can issue complete SCSI command to FC-SCSI HBA(Fibre Channel SCSI Host Bus Adapter) assigned to the guest domain.
Using this functionality, guest domain can control such as tape

device

Quoted 11 lines

for backup and so on.

We hope these patches will be merged into Xen 3.2.0.

Thanks,

-----
Jun Kamada

Quoted 4 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Quoted 1 line

This seems to have been hanging around for ages. If there's no

argument

Quoted 1 line

otherwise I'll give it a read through and stick it in for 3.2.0.

Yep, its certainly a candidate for 3.2, but I'd class the API as "experimental" at this stage and not commit to supporting it going forward as-is just yet.

Things I'd like to see:
* performance results (hopefully at the summit!)

* variable sized comms rings (fixed at connect time) so we can queue
more descriptors if req'd.
* support for pass through of individual scsi logical units to the
dom0's linux scsi-generic layer rather than whole HBAs -- I think most users will want to hide all that FC nastiness and just present a LUN.

Hi all,

I will send total five patches on following e-mails.

These patches provide functionality that guest domain can issue complete SCSI command to FC-SCSI HBA(Fibre Channel SCSI Host Bus Adapter) assigned to the guest domain.
Using this functionality, guest domain can control such as tape device for backup and so on.

We hope these patches will be merged into Xen 3.2.0.

I am very interested in this functionality, but mainly from a Windows point of view (doing test restores of backups for customers). What would be involved in getting this working under a HVM domain in the absence of PV drivers?

Failing that, I'll see if I can incorporate it into the Windows PV drivers I'm working on...

Hi all,

I will send total five patches on following e-mails.

These patches provide functionality that guest domain can issue complete SCSI command to FC-SCSI HBA(Fibre Channel SCSI Host Bus Adapter) assigned to the guest domain.
Using this functionality, guest domain can control such as tape device for backup and so on.

We hope these patches will be merged into Xen 3.2.0.

Thanks,

-----
Jun Kamada

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

What's the CONFIG_XEN_FC stuff all about? It doesn't appear to be configurable (so it's always off), it's ifdef'ed all over the place, and there are dead files (e.g., fcfront.c) that presumably would depend on that option.

Either it needs to be plumbed in properly, or the XEN_FC stuff needs to be stripped out entirely from this patchset.

The CONFIG_XEN_FC stuff is used for activating FC(Fibre Channel) transport layer functionality. I will attach additional patch for the stuff.

Another question about the original patches: did you test that a 64-bit backend can communicate with 32-bit frontend (and vice versa)? Either you need to make sure that all the shared-memory structures defined in vscsiif.h compile identically for 32-bit and 64-bit mode, or you need to detect the frontend's 'bitness' in scsiback and optionally do 32-to-64 or 64-to-32 conversion. I think it's obvious which is less hassle. ;-)

Handling this is really a prerequisite for putting these patches into a stable Xen release.

Quoted 6 lines

Another question about the original patches: did you test that a 64-bit backend can communicate with 32-bit frontend (and vice versa)? Either you need to make sure that all the shared-memory structures defined in vscsiif.h compile identically for 32-bit and 64-bit mode, or you need to detect the frontend's 'bitness' in scsiback and optionally do 32-to-64 or 64-to-32 conversion. I think it's obvious which is less hassle. ;-)

I suppose that the VBD uses the later method. (Is it right?) However, the later method requires 'optimize level of frontend on compile time' in addition to the 'bitness'. If the frontend is optimized for space, members in structure are packed. On the other hand, if the frontend is optimized for speed, each members are aligned on word boundary. (padding will be inserted.)
How do you think about this?

Anyway, the right answer is to ensure that your structures compile the same whether compiled with 32-bit or 64-bit gcc. Check the structure sizes, fields sizes and field offsets.

Quoted 3 lines

Anyway, the right answer is to ensure that your structures compile the same whether compiled with 32-bit or 64-bit gcc. Check the structure sizes, fields sizes and field offsets.

You showed me two options in previous email; i.) all the shared-memory structures defined in vscsiif.h compile identically for 32-bit and 64-bit mode, ii.) to detect the frontend's 'bitness' in scsiback and optionally do 32-to-64 or 64-to-32 conversion.

I consider that current VBD implementation takes option ii.), however you recommended me to take option i.) for pvSCSI driver. Is my understanding right?

Quoted 9 lines · Jun Kamada

Quoted 3 lines

Anyway, the right answer is to ensure that your structures compile the same whether compiled with 32-bit or 64-bit gcc. Check the structure sizes, fields sizes and field offsets.

You showed me two options in previous email; i.) all the shared-memory structures defined in vscsiif.h compile identically for 32-bit and 64-bit mode, ii.) to detect the frontend's 'bitness' in scsiback and optionally do 32-to-64 or 64-to-32 conversion.

Quoted 3 lines

I consider that current VBD implementation takes option ii.), however you recommended me to take option i.) for pvSCSI driver. Is my understanding right?

Yes. It was an accident we ended up with structures compiling differently on 32- vs 64-bit. By the time we discovered it was an issue, we couldn't break backward compatibility.

Keir-san,

Quoted 11 lines · On Wed, 24 Oct 2007 08:42:38 +0100 Keir Fraser <[email protected]> wrote:

Quoted 9 lines · Jun Kamada

Quoted 3 lines

Anyway, the right answer is to ensure that your structures compile the same whether compiled with 32-bit or 64-bit gcc. Check the structure sizes, fields sizes and field offsets.

You showed me two options in previous email; i.) all the shared-memory structures defined in vscsiif.h compile identically for 32-bit and 64-bit mode, ii.) to detect the frontend's 'bitness' in scsiback and optionally do 32-to-64 or 64-to-32 conversion.

Quoted 7 lines

Quoted 3 lines

I consider that current VBD implementation takes option ii.), however you recommended me to take option i.) for pvSCSI driver. Is my understanding right?

Yes. It was an accident we ended up with structures compiling differently on 32- vs 64-bit. By the time we discovered it was an issue, we couldn't break backward compatibility.

I understood. I'll try to modify the pvSCSI driver by option i.).

Best regards,

-----
Jun Kamada

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Keir-san,

I have finished the modification of the driver by option i.). The pvSCSI driver can work on heterogeneous (x86_32 and x86_64) environment.

I will attach total six patches on following mail.

Best regards,

Quoted 33 lines · On Thu, 25 Oct 2007 10:59:31 +0900 Jun Kamada <[email protected]> wrote:

Keir-san,

Quoted 12 lines · On Wed, 24 Oct 2007 08:42:38 +0100 Keir Fraser <[email protected]> wrote:

Quoted 10 lines · Jun Kamada

Quoted 4 lines

Anyway, the right answer is to ensure that your structures compile the same
whether compiled with 32-bit or 64-bit gcc. Check the structure sizes, fields sizes and field offsets.

You showed me two options in previous email; i.) all the shared-memory structures defined in vscsiif.h compile identically for 32-bit and 64-bit mode, ii.) to detect the frontend's 'bitness' in scsiback and optionally do 32-to-64 or 64-to-32 conversion.

Quoted 7 lines

Quoted 3 lines

I consider that current VBD implementation takes option ii.), however you recommended me to take option i.) for pvSCSI driver. Is my understanding right?

Yes. It was an accident we ended up with structures compiling differently on 32- vs 64-bit. By the time we discovered it was an issue, we couldn't break backward compatibility.

I understood. I'll try to modify the pvSCSI driver by option i.).

Best regards,

-----
Jun Kamada

Quoted 4 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

-----
Jun Kamada

Vscsiif.h still needs clean up:
1. You cannot have Linux-specific stuff in that header file. It's supposed
to be generic.

2. You cannot pollute the global namespace with unqualified names. All your
names, macros, etc should have a vscsiif_ or VSCSIIF_ prefix. Names like DEFAULT_CAN_QUEUE and SG_TABLESIZE pollute the namespace and it's not immediately obvious which header they come from (which it would be if they were prefixed).

3. Do you really mean for vscsiif_btf_response to be empty? And neither
response structure has an identifier to match the request it responds to. Can requests not be handled out of order by your drivers? That would seem very odd.

Also CONFIG_XEN_FC is ifdef'ed all over scsifront and scsiback. You'll have to find some way to clean that up, perhaps by providing null stub implementations of fc functions in a header file if !CONFIG_XEN_FC.

FIXME comments in the connection setup code, accompanied by hard-coded constants, don't bode well. Is the connection setup protocol fully baked yet?

All this, coupled with the fact that you can only export whole HBAs, which doesn't seem likely to be a very popular usage scenario, makes me think this driver should wait until after 3.2.0.

Quoted 3 lines

All this, coupled with the fact that you can only export whole HBAs, which doesn't seem likely to be a very popular usage scenario, makes me think this driver should wait until after 3.2.0.

We consider that SCSI support on guest domain is mandatory for bussiness use. Using a NPIV technology on host domain, each guest domain can have own virtual HBA. We have already evaluated on our environment the conbination of NPIV and pvSCSI work well.
On the other hand, we undestand the usefullness of LUN assignment to guest domain. We would like to develop it on next step in our current plan.

Quoted 11 lines · Jun Kamada

Quoted 3 lines

All this, coupled with the fact that you can only export whole HBAs, which doesn't seem likely to be a very popular usage scenario, makes me think this driver should wait until after 3.2.0.

We consider that SCSI support on guest domain is mandatory for bussiness use. Using a NPIV technology on host domain, each guest domain can have own virtual HBA. We have already evaluated on our environment the conbination of NPIV and pvSCSI work well.
On the other hand, we undestand the usefullness of LUN assignment to guest domain. We would like to develop it on next step in our current plan.

Okay, well re-spin the patches and I'll take another look.

Quoted 1 line

Okay, well re-spin the patches and I'll take another look.

Can I ask one question in order to avoid my misunderstanding?

Do you mean that the pvSCSI driver is never commited until it supports LUN assignment to geust domain, or anything else?

Hi,

On Tue, 2007-10-30 at 19:39 +0900, Jun Kamada wrote:

Is this using the same syntax for HBA reservation that we used to

ie. simply enumerated using the scsi host number on the dom0?

If so, that is rather fragile. The scsi host number can change from kernel to kernel depending on HBA driver probe order and PCI bus enumeration order. It changes when hardware is rearranged. And how

we deal with migrate, when we'll need to select a completely different HBA on a different host?

Selecting the HBA by scsi host number is good enough for a

that the code works, but I don't think it's going to be robust enough for production deployment.

Also, has there been any progress towards exposing individual LUNs to the guest rather than a whole HBA?

Quoted 5 lines

Hi,

On Tue, 2007-10-30 at 19:39 +0900, Jun Kamada wrote:

Is this using the same syntax for HBA reservation that we used to

have,

Quoted 5 lines

ie. simply enumerated using the scsi host number on the dom0?

If so, that is rather fragile. The scsi host number can change from kernel to kernel depending on HBA driver probe order and PCI bus enumeration order. It changes when hardware is rearranged. And how

do

Quoted 4 lines

we deal with migrate, when we'll need to select a completely different HBA on a different host?

Selecting the HBA by scsi host number is good enough for a

demonstration

Quoted 3 lines

that the code works, but I don't think it's going to be robust enough for production deployment.

Quoted 2 lines

Also, has there been any progress towards exposing individual LUNs to the guest rather than a whole HBA?

People keep saying individual LUN... do they actually mean SCSI Target devices? Splitting up devices on the same HBA should be reasonably simple, but due to the way the scsi protocol works (AFAIK), splitting up LUN's on a single SCSI device could be a bit trickier...

Or maybe LUN doesn't mean what I think it means in this case... my understanding is:

HBA = Host Bus Adapter (eg a PCI SCSI adapter)
SCSI Target Device = one of the physical devices that sits on the SCSI bus (eg a tape drive or a disk drive)
LUN (Logical Unit Number) = a sub unit of the SCSI Target Device, eg a single disk drive in a RAID array, a tape device in an multi tape drive autoloader, a CD ROM in a CDROM tower.

On Tue, 2007-10-30 at 19:39 +0900, Jun Kamada wrote:

Is this using the same syntax for HBA reservation that we used to have, ie. simply enumerated using the scsi host number on the dom0?

If so, that is rather fragile. The scsi host number can change from kernel to kernel depending on HBA driver probe order and PCI bus enumeration order. It changes when hardware is rearranged. And how do we deal with migrate, when we'll need to select a completely different HBA on a different host?

Selecting the HBA by scsi host number is good enough for a demonstration that the code works, but I don't think it's going to be robust enough for production deployment.

Also, has there been any progress towards exposing individual LUNs to the guest rather than a whole HBA?

Cheers,
Stephen

We also recognize that more sophisticated interface is needed. WWPN can be used for FC-SCSI, however pure SCSI does not have any unique ID. Do you have any idea?

Hi,

Quoted 3 lines · Jun Kamada

We also recognize that more sophisticated interface is needed. WWPN can be used for FC-SCSI, however pure SCSI does not have any unique ID. Do you have any idea?

Well... Linux does try to use address-by-content to some extent, but that's going to be fragile here, as well. It's complicated even more by the difficulty of having potential multipath in the mix.

libblkid has some helpful functions for certain cases, but that doesn't help at all for the bootstrap cases when you're dealing with completely unformatted disks.

The libvirt folks have been wrestling with some of the same questions recently, libvirt-devel may have some useful discussion.

--Stephen

Hi,

On Tue, 2007-10-30 at 19:39 +0900, Jun Kamada wrote:

Is this using the same syntax for HBA reservation that we used to have, ie. simply enumerated using the scsi host number on the dom0?

If so, that is rather fragile. The scsi host number can change from kernel to kernel depending on HBA driver probe order and PCI bus enumeration order. It changes when hardware is rearranged. And how do we deal with migrate, when we'll need to select a completely different HBA on a different host?

Selecting the HBA by scsi host number is good enough for a demonstration that the code works, but I don't think it's going to be robust enough for production deployment.

Also, has there been any progress towards exposing individual LUNs to the guest rather than a whole HBA?

Cheers,
Stephen

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel