Skip to contentSkip to conversation

pv-scsi driver (scsiback/scsifront)

RFC In review Tomonari Horikoshi 4 participants 6 messages May 16, 2007 – May 22, 2007

lore
Share & tools
lore

Tomonari Horikoshi

author
#0 lore raw

RFC 1/1 pv-scsi driver (scsiback/scsifront)

Hi all.

We developped a pv-scsi driver that we refered Fujita-san's scsi-driver and blkback.
(see, http://www.xensource.com/files/xensummit_4/Xen_Summit_8_Matsumoto.pdf)

The pv-scsi driver's feature is as follow:
* Guest has dedicated SCSI-HBAs of Dom0.
* Guest can send scsi_cdb to the HBAs.
* Guest recognises the HBAs from hostno of xenstore.

Currentlly, We are developping FC version based on this.

* Future work:
* implement python code
* performance tunning
* attach, detach
* suspend, resume

* We are wondering about:
* We used "scsihost" as xenstore nodename. Is it suitable?
* We consider about configfile format...
scsihost = ['fc,0', 'scsi,1', 'type,num']
type = "fc" or "scsi"
num = scsi host number on Dom0
Do you have any comment?
* We have no idea how to implement suspend/resume feature. ex. Physical HBA mapping for resumed guest. Pending I/O.
The WWN within FC mode for resumed guest. Influence of migration.
...

Could you suggest to us about this?

Best regards,
Tomonari Horikoshi

# HG changeset patch
# User [email protected]
# Date 1179302059 -32400
# Node ID 6913be6b4f539aaf8a7ed6deb88247843eb88e2f # Parent d4a0706d6747cd54613328fe630c3bcb7cafbac5 add scsiback driver

Signed-off-by Tomonari Horikoshi <[email protected]>
Signed-off-by Tsunehisa Doi <[email protected]>
Signed-off-by Jun Kamada <[email protected]>
Signed-off-by Akira Hayakawa <[email protected]>
linux-2.6-xen-sparse/drivers/xen/scsiback/Makefile new file +2 -0
diff -r d4a0706d6747 -r 6913be6b4f53 linux-2.6-xen-sparse/drivers/xen/scsiback/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_XEN_SCSI_BACKEND)	+= scsibk.o
+scsibk-y			+= interface.o scsiback.o
linux-2.6-xen-sparse/drivers/xen/scsiback/common.h new file +103 -0

103 lines not on this page · raw message

linux-2.6-xen-sparse/drivers/xen/scsiback/interface.c new file +165 -0

165 lines not on this page · raw message

linux-2.6-xen-sparse/drivers/xen/scsiback/scsiback.c new file +716 -0

716 lines not on this page · raw message

xen/include/public/io/scsiif.h new file +49 -0

49 lines not on this page · raw message

# HG changeset patch
# User [email protected]
# Date 1179302161 -32400
# Node ID 38b476ccc82814fb55e3a36f88c995e353cf63dc # Parent 6913be6b4f539aaf8a7ed6deb88247843eb88e2f add scsifront driver

Signed-off-by Tomonari Horikoshi <[email protected]>
Signed-off-by Tsunehisa Doi <[email protected]>
Signed-off-by Jun Kamada <[email protected]>
Signed-off-by Akira Hayakawa <[email protected]>
linux-2.6-xen-sparse/drivers/xen/scsifront/Makefile new file +4 -0
diff -r 6913be6b4f53 -r 38b476ccc828 linux-2.6-xen-sparse/drivers/xen/scsifront/Makefile
@@ -0,0 +1,4 @@
+obj-$(CONFIG_XEN_SCSI_FRONTEND)	:= xenscsi.o
+
+xenscsi-objs := scsifront.o
+
linux-2.6-xen-sparse/drivers/xen/scsifront/scsifront.c new file +524 -0

524 lines not on this page · raw message

# HG changeset patch
# User [email protected]
# Date 1179302318 -32400
# Node ID 7a02a4017780fc2d97232572d9e7d88a02ca9b9c # Parent 38b476ccc82814fb55e3a36f88c995e353cf63dc modiy config

Signed-off-by Tomonari Horikoshi <[email protected]>
Signed-off-by Tsunehisa Doi <[email protected]>
Signed-off-by Jun Kamada <[email protected]>
Signed-off-by Akira Hayakawa <[email protected]>
buildconfigs/linux-defconfig_xen_ia64 +2 -0
diff -r 38b476ccc828 -r 7a02a4017780 buildconfigs/linux-defconfig_xen_ia64
@@ -1630,6 +1630,7 @@ CONFIG_XEN_XENBUS_DEV=y
 CONFIG_XEN_XENBUS_DEV=y
 CONFIG_XEN_BACKEND=y
 CONFIG_XEN_BLKDEV_BACKEND=y
+CONFIG_XEN_SCSI_BACKEND=y
 CONFIG_XEN_BLKDEV_TAP=y
 CONFIG_XEN_NETDEV_BACKEND=y
 # CONFIG_XEN_NETDEV_PIPELINED_TRANSMITTER is not set

@@ -1641,6 +1642,7 @@ CONFIG_XEN_PCIDEV_BACKEND_SLOT=y
 # CONFIG_XEN_PCIDEV_BE_DEBUG is not set
 CONFIG_XEN_TPMDEV_BACKEND=m
 CONFIG_XEN_BLKDEV_FRONTEND=y
+CONFIG_XEN_SCSI_FRONTEND=m
 CONFIG_XEN_NETDEV_FRONTEND=y
 CONFIG_XEN_FRAMEBUFFER=y
 CONFIG_XEN_KEYBOARD=y
buildconfigs/linux-defconfig_xen_x86_32 +4 -2
diff -r 38b476ccc828 -r 7a02a4017780 buildconfigs/linux-defconfig_xen_x86_32
@@ -1083,13 +1083,13 @@ CONFIG_IDEDMA_AUTO=y
 # SCSI device support
 #
 CONFIG_RAID_ATTRS=m
-CONFIG_SCSI=m
+CONFIG_SCSI=y
 CONFIG_SCSI_PROC_FS=y
 
 #
 # SCSI support type (disk, tape, CD-ROM)
 #
-CONFIG_BLK_DEV_SD=m
+CONFIG_BLK_DEV_SD=y
 CONFIG_CHR_DEV_ST=m
 CONFIG_CHR_DEV_OSST=m
 CONFIG_BLK_DEV_SR=m

@@ -3253,6 +3253,7 @@ CONFIG_XEN_XENBUS_DEV=y
 CONFIG_XEN_XENBUS_DEV=y
 CONFIG_XEN_BACKEND=y
 CONFIG_XEN_BLKDEV_BACKEND=y
+CONFIG_XEN_SCSI_BACKEND=y
 CONFIG_XEN_BLKDEV_TAP=y
 CONFIG_XEN_NETDEV_BACKEND=y
 # CONFIG_XEN_NETDEV_PIPELINED_TRANSMITTER is not set

@@ -3264,6 +3265,7 @@ CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
 # CONFIG_XEN_PCIDEV_BE_DEBUG is not set
 CONFIG_XEN_TPMDEV_BACKEND=m
 CONFIG_XEN_BLKDEV_FRONTEND=y
+CONFIG_XEN_SCSI_FRONTEND=m
 CONFIG_XEN_NETDEV_FRONTEND=y
 CONFIG_XEN_FRAMEBUFFER=y
 CONFIG_XEN_KEYBOARD=y
linux-2.6-xen-sparse/drivers/xen/Kconfig +16 -0
diff -r 38b476ccc828 -r 7a02a4017780 linux-2.6-xen-sparse/drivers/xen/Kconfig
@@ -53,6 +53,14 @@ config XEN_BLKDEV_BACKEND
 	  The block-device backend driver allows the kernel to export its
 	  block devices to other guests via a high-performance shared-memory
 	  interface.
+
+config XEN_SCSI_BACKEND
+       tristate "SCSI backend driver"
+       depends on XEN_BACKEND
+       default y
+       help
+         The SCSI backend driver allows the kernel to export its SCSI HBAs
+         to other guests via a high-performance shared-memory interface.
 
 config XEN_BLKDEV_TAP
 	tristate "Block-device tap backend driver"

@@ -162,6 +170,14 @@ config XEN_BLKDEV_FRONTEND
 	  dedicated device-driver domain, or your master control domain
 	  (domain 0), then you almost certainly want to say Y here.
 
+config XEN_SCSI_FRONTEND
+	tristate "SCSI frontend driver"
+	depends on XEN && SCSI
+	default y
+	help
+	  The SCSI frontend driver allows the kernel to access SCSI HBAs
+	  within another guest OS.
+
 config XEN_NETDEV_FRONTEND
 	tristate "Network-device frontend driver"
 	depends on XEN && NET
linux-2.6-xen-sparse/drivers/xen/Makefile +2 -0
diff -r 38b476ccc828 -r 7a02a4017780 linux-2.6-xen-sparse/drivers/xen/Makefile
@@ -9,10 +9,12 @@ obj-y	+= char/
 
 obj-y	+= util.o
 obj-$(CONFIG_XEN_BLKDEV_BACKEND)	+= blkback/
+obj-$(CONFIG_XEN_SCSI_BACKEND)		+= scsiback/
 obj-$(CONFIG_XEN_BLKDEV_TAP)		+= blktap/
 obj-$(CONFIG_XEN_NETDEV_BACKEND)	+= netback/
 obj-$(CONFIG_XEN_TPMDEV_BACKEND)	+= tpmback/
 obj-$(CONFIG_XEN_BLKDEV_FRONTEND)	+= blkfront/
+obj-$(CONFIG_XEN_SCSI_FRONTEND)		+= scsifront/
 obj-$(CONFIG_XEN_NETDEV_FRONTEND)	+= netfront/
 obj-$(CONFIG_XEN_PCIDEV_BACKEND)	+= pciback/
 obj-$(CONFIG_XEN_PCIDEV_FRONTEND)	+= pcifront/

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#1 lore raw
Quoted 19 lines · Tomonari Horikoshi

Hi all.

We developped a pv-scsi driver that we refered Fujita-san's scsi-driver and blkback.
(see, http://www.xensource.com/files/xensummit_4/Xen_Summit_8_Matsumoto.pdf)

The pv-scsi driver's feature is as follow:
* Guest has dedicated SCSI-HBAs of Dom0.
* Guest can send scsi_cdb to the HBAs.
* Guest recognises the HBAs from hostno of xenstore.

Currentlly, We are developping FC version based on this.

* Future work:
* implement python code
* performance tunning
* attach, detach
* suspend, resume

Quoted 6 lines

* We are wondering about:
* We used "scsihost" as xenstore nodename. Is it suitable?
* We consider about configfile format...
scsihost = ['fc,0', 'scsi,1', 'type,num']
type = "fc" or "scsi"
num = scsi host number on Dom0

I would expect "fc" does not need to be specified, unless there is FC-isms exposed to the guest.

Quoted 5 lines

Do you have any comment?
* We have no idea how to implement suspend/resume feature. ex. Physical HBA mapping for resumed guest. Pending I/O.
The WWN within FC mode for resumed guest.

The WWN is a whole different issue - and I'm going to want to make sure that whatever you do here is consistent with FC NPIV virtual ports instantiated in Dom 0. See:
http://marc.info/?l=linux-scsi&m=117768770720886&w=2

Quoted 18 lines

Influence of migration.
...

Could you suggest to us about this?

Best regards,
Tomonari Horikoshi

------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#2 lore raw

Thank you for your comment.

I work in same group Tomonari Horikoshi works.

Quoted 21 lines
Quoted 20 lines · Tomonari Horikoshi

Hi all.

We developped a pv-scsi driver that we refered Fujita-san's scsi-driver and blkback.
(see, http://www.xensource.com/files/xensummit_4/Xen_Summit_8_Matsumoto.
pdf)

The pv-scsi driver's feature is as follow:
* Guest has dedicated SCSI-HBAs of Dom0.
* Guest can send scsi_cdb to the HBAs.
* Guest recognises the HBAs from hostno of xenstore.

Currentlly, We are developping FC version based on this.

* Future work:
* implement python code
* performance tunning
* attach, detach
* suspend, resume

Quoted 8 lines
Quoted 6 lines

* We are wondering about:
* We used "scsihost" as xenstore nodename. Is it suitable?
* We consider about configfile format...
scsihost = ['fc,0', 'scsi,1', 'type,num']
type = "fc" or "scsi"
num = scsi host number on Dom0

I would expect "fc" does not need to be specified, unless there is FC-isms exposed to the guest.

We want to use SAN management software on guest OS. The software works on native(no VM) linux. So we think it is necesarry to have guest OS shown whether HBA card is FC or SCSI in the same way of native linux.

Quoted 9 lines
Quoted 5 lines

Do you have any comment?
* We have no idea how to implement suspend/resume feature. ex. Physical HBA mapping for resumed guest. Pending I/O.
The WWN within FC mode for resumed guest.

The WWN is a whole different issue - and I'm going to want to make sure that whatever you do here is consistent with FC NPIV virtual ports instantiated in Dom 0. See:
http://marc.info/?l=linux-scsi&m=117768770720886&w=2

We think whether WWN is same value or not when a guest resumes again is unknown because the WWN may be already used by another guest.

Best Regards,
Akira Hayakawa

#3 lore raw
Quoted 7 lines
Quoted 3 lines

I would expect "fc" does not need to be specified, unless there is FC-isms exposed to the guest.

We want to use SAN management software on guest OS. The software
works on native(no VM) linux. So we think it is necesarry to have guest OS shown whether HBA card is FC or SCSI in the same way of native linux.

Well - depends on what/how your san mgmt works. If it's straight scsi, then it would be fine - but you can't talk to anything non-scsi and not enumerated by the hba. If it's layered on hbaapi, it does mean you want to talk FC, not just scsi, and now things change significantly.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Do you have any comment?
* We have no idea how to implement suspend/resume feature. ex. Physical HBA mapping for resumed guest. Pending I/O.
The WWN within FC mode for resumed guest.

Quoted 7 lines
Quoted 4 lines

The WWN is a whole different issue - and I'm going to want to make sure that whatever you do here is consistent with FC NPIV virtual ports instantiated in Dom 0. See:
http://marc.info/?l=linux-scsi&m=117768770720886&w=2

We think whether WWN is same value or not when a guest resumes again is unknown because the WWN may be already used by another guest.

This confuses me greatly. WWN's are how FC ports are known - which controls their SAN visibility and device access. If it's changing for the VM, unless you have everything seeing everything (i.e. no SAN zoning or lun masking, which is very very rare in a production environment) then whether you see your storage is questionable. For this reason, regular NPIV will be adding the WWNs as a resource of the guest, much like the ethernet MAC addresses. And, if it is bound to the guest, it matches the model needed for suspend/resume, although there are challenges for discovery and enumeration.

Additionally, I certainly hope you are keeping far more control on how WWN's are allocated and used. There is that small part about uniqueness that has to be maintained or the fabric will show very nasty issues.

-- james

#4 lore raw
In reply to Tomonari Horikoshi
Quoted 1 line

We developped a pv-scsi driver that we refered Fujita-san's

scsi-driver

Quoted 3 lines

and blkback.
(see,
http://www.xensource.com/files/xensummit_4/Xen_Summit_8_Matsumoto.pdf)

This is good work, and we'd certainly like to get it polished and in mainline -- thanks!

Quoted 2 lines

The pv-scsi driver's feature is as follow:
* Guest has dedicated SCSI-HBAs of Dom0.

Is it really the case that you must dedicate a HBA to the guest? Surely we can extend it to enable an individual LUN to be mapped through to a guest, translating the host:bus:id etc accordingly?

Quoted 3 lines

* We consider about configfile format...
scsihost = ['fc,0', 'scsi,1', 'type,num']
type = "fc" or "scsi"

Why do you need to select between fc and scsi?

Quoted 7 lines

* We have no idea how to implement suspend/resume feature. ex. Physical HBA mapping for resumed guest. Pending I/O.
The WWN within FC mode for resumed guest. Influence of migration.

Could you suggest to us about this?

The blkfront/back code is obviously a good crib for this. Basically, the front end driver needs to store enough information to be able to reissue any uncompleted requests across a migration. This is accomplished with a 'shadow ring'. The frontend needs to be capable of reconnecting if the backend goes out of state connected, and then reissue the requests. For migration write-after-write safety, the backend shouldn't close until all outstanding requests have either been completed or aborted.

One other thing I notice is that you've kept the blk ring protocol's 11 page limit per request. The blkring kinda gets away with this because (at least in principle) we could merge consecutive requests in blkback to create larger IOs. I guess we could do that with scsi requests too, but I'd feel more comfortable if we didn't mess with the request stream that the guest is generating. We probably need to make the number of pages in an SG list variable, and hence have variable sized requests across the ring. We should defintiely make the ring multi-page too.

What do you think?

Thanks,
Ian

#5 lore raw

Thank you for your comment.
I'm sorry for delaying response.

I work in same group Tomonari Horikoshi works.

Quoted 6 lines
Quoted 1 line

We developped a pv-scsi driver that we refered Fujita-san's

scsi-driver

Quoted 3 lines

and blkback.
(see,
http://www.xensource.com/files/xensummit_4/Xen_Summit_8_Matsumoto.pdf)

Quoted 7 lines

This is good work, and we'd certainly like to get it polished and in mainline -- thanks!

Quoted 2 lines

The pv-scsi driver's feature is as follow:
* Guest has dedicated SCSI-HBAs of Dom0.

Is it really the case that you must dedicate a HBA to the guest?

Yes, and we are planning to use NPIV function so that more than one guest can use a HBA.

Quoted 8 lines

Surely we can extend it to enable an individual LUN to be mapped through to a guest, translating the host:bus:id etc accordingly?

Quoted 3 lines

* We consider about configfile format...
scsihost = ['fc,0', 'scsi,1', 'type,num']
type = "fc" or "scsi"

Why do you need to select between fc and scsi?

Please refer "Re: [RFC] pv-scsi driver(scsiback/scsifront)"(respose to [email protected].).

Quoted 8 lines
Quoted 7 lines

* We have no idea how to implement suspend/resume feature. ex. Physical HBA mapping for resumed guest. Pending I/O.
The WWN within FC mode for resumed guest. Influence of migration.

Could you suggest to us about this?

Quoted 7 lines

The blkfront/back code is obviously a good crib for this. Basically, the front end driver needs to store enough information to be able to reissue any uncompleted requests across a migration. This is accomplished with a 'shadow ring'. The frontend needs to be capable of reconnecting if the backend goes out of state connected, and then reissue the requests. For migration write-after-write safety, the backend shouldn't close until all outstanding requests have either been completed or aborted.

Thank you so much, your advise is very helpful to us.

Quoted 10 lines

One other thing I notice is that you've kept the blk ring protocol's 11 page limit per request. The blkring kinda gets away with this because (at least in principle) we could merge consecutive requests in blkback to create larger IOs. I guess we could do that with scsi requests too, but I'd feel more comfortable if we didn't mess with the request stream that the guest is generating. We probably need to make the number of pages in an SG list variable, and hence have variable sized requests across the ring. We should defintiely make the ring multi-page too.

What do you think?

I agree with you. We try to defintiely make the ring multi-page.

Quoted 2 lines

Thanks,
Ian

Best Regards,
Aki