SCSI target for IBM Power5 LPAR
Subsystems (3)
RFC 1/1 SCSI target for IBM Power5 LPAR
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.
One of the biggest discussions, I expect, will be whether much of this belongs in user-land. We have had considerable debates on that subject, and ended up with it all in the kernel. Unlike some other targets (e.g. iSCSI) that sit on top of nice TCP/IP stacks, this one interacts at a pretty low level with the firmware. There weren't any good user-land splits that seemed to make sense (and looking at the history of the NFS server also influenced the discussion.)
This is currently just an RFC...if my boxers survive the flamage I'll submit it for inclusion.
Thanks
Dave B
drivers/scsi/ibmvscsi/ibmvscsi.h +1 -0
@@ -33,6 +33,7 @@#include <linux/list.h> #include <linux/completion.h> #include <linux/interrupt.h> +#include <scsi/scsi_cmnd.h> #include "viosrp.h" struct scsi_cmnd ;
drivers/scsi/ibmvscsi/ibmvscsis.c +3775 -0
2,993 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Hi Dave,
Quoted 4 lines
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (
drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.
Good stuff. Got a couple of small suggestions.
+/* Allocate a buffer with a dma_address. Don't use dma_alloc_coherent
+ * since that uses GFP_ATOMIC internally and we can tollerate a delay
+ */
+static void *alloc_coherent_buffer(struct server_adapter *adapter, size_t size,
+ dma_addr_t *dma_handle)
+{
+ void *buffer = kmalloc(size, GFP_KERNEL);
+
+ if (buffer) {
+ *dma_handle = dma_map_single(adapter->dev, buffer, size,
+ DMA_BIDIRECTIONAL);
+
+ if (dma_mapping_error(*dma_handle)) {
+ kfree(buffer);
+ buffer = NULL;
+ }
+ }
+
+ return buffer;
+}
This should be fixed in mainline, on ppc64 we no longer build the dma_* ops on top of the pci_* ops. This means we actually look at the flags :)
+ adapter->max_sectors = MAX_SECTORS;
Does this mean we are limited to 128kB transfers? Would it be OK to bump the default?
Anton
Hi Anton,
Quoted 4 lines
drivers/scsi/ibmvscsi/ibmvscsis.c +1
-0+ adapter->max_sectors =MAX_SECTORS ;Does this mean we are limited to 128kB transfers? Would it be OK to bump the default?
We use MAX_SECTORS (which is actually 127.5kB) because that's the max_sectors of the loopback device (we have a lot of users that like the flexibility of using loopback with the ibmvscsis driver)... It can be bumped up without any problems because there is code that splits requests if they are larger than the target's max_sectors...
What would you recommend? 256kB?
Signature · 3 lines
Santiago A. Leon Power Linux Development IBM Linux Technology Center
Hi,
Quoted 7 lines
We use
MAX_SECTORS(which is actually 127.5kB) because that's themax_sectorsof the loopback device (we have a lot of users that like the flexibility of using loopback with the ibmvscsis driver)... It can be bumped up without any problems because there is code that splits requests if they are larger than the target'smax_sectors...What would you recommend? 256kB?
~256kB sounds reasonable to me and I notice that other VIO server is setting it to 256kB :)
Anton
Quoted 4 lines · Dave C Boutcher
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (
drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.
Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (
drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.
There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.
Vlad, can you comment on the state of stgt and whether you see it being ready for mainline any time soon?
Signature · 1 line
Dave Boutcher
From: Dave C Boutcher <[email protected]>
Subject: Re: [RFC] SCSI target for IBM Power5 LPAR
Date: Wed, 7 Sep 2005 07:45:04 -0500
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Dave C Boutcher
Quoted 4 lines
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (
drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.
Quoted 6 lines
Quoted 2 lines
Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.
The generic scsi target framework (stgt) is not SCST (that Vlad has maintained). It is the project that Mike Christie and I started last month. We discussed it with Christoph and decided that it would be better to start from scratch because of the design differences.
Quoted 2 lines · FUJITA Tomonori
month. We discussed it with Christoph and decided that it would be better to start from scratch because of the design differences.
Some of the things we are trying to improve upon are things that are better supported in 2.6. Some differences:
- We will support controller hotplug.
- We allow any type of device (dm, scsi, ide, LVM@ etc) as storage. And we do not want hook into SCSI-ml's upper layer drivers and deal with that refcounting if we can help it so we push a lot of code to userspace and only do reads and writes in the kernel.
- We also hope to support any block layer target.
- As mentioned before, scatterlists by using the block layer's support.
There may be more that I am forgetting, but originally we started out by trying to clean up the SCST code for 2.6 and make it resemble SCSI-ml's hotplug model. As we did this it looked like some code could live in userspace and it would end up being a rewrite becuase there was so much to do so we started a new project. We hope to work with Vlad and the SCST developers.
Re: remove-single-device removes mounted HDDs (kernel 2.6)
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Harald Seipp
Christoph Hellwig wrote on 11.08.2005 18:40:03:
Quoted 5 lines · Harald Seipp
Quoted 3 lines
Thank you for your quick responses! It finally looks like the task I need to do - fully rescan all hosts for newly attached/changed devices - must include the logic to skip devices with mounted fs for 2.6.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Why do you want to rescan a full bus anyway? What SCSI transport do you want this for? All transports where new targets can be hot-plugged should be doing automatic scanning already.
Old-fashioned parallel SCSI bus and FC/SAN where there are no hot-plug events AFAIK.
In FC you get per-target events, although none for new LUs showing up on a given target (in fact no SAM transport does the latter)
I believe some Fibre Channel storage arrays generate RSCN's when new logical units are created (although I don't know if the existing Linux Fibre Channel HBA drivers propagate these registered state change notifications to the hot-plug subsystem).
I understand that many iSCSI storage arrays will generate a SCSI Asynchronous Event Notification for the REPORTED LUNS DATA HAS CHANGED Unit Attention condition that SPC-3 says shall be generated when a target changes its logical unit inventory, provided that AEN's are enabled in the control mode page.
So there are SAM transports that generate events when a new LU shows up.
I'm curious as to what plans, if any, exist for handling these events in Linux.
Regards,
-Steve
--------
Steve Byan <[email protected]>
Software Architect
Egenera, Inc.
165 Forest Street
Marlboro, MA 01752
(508) 858-3125
Quoted 9 lines · Dave C Boutcher
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (
drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.
Quoted 8 lines
Quoted 2 lines
Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.
stgt is not scst. It is new. And yes we will have proper support for sccatterlists. We need something similar to what sg and st need so I plan on using the block layer functions that are implemented for those upper layer drivers for stgt. If you want to help out and implement that then that would be good. We are also very early in our design so all comments are welcome.
Quoted 13 lines · Dave C Boutcher
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (
drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.
Quoted 6 lines
There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.
Vlad, can you comment on the state of stgt and whether you see it being ready for mainline any time soon?
Sorry, I can see on stgt page only mail lists archive and not from start (from Aug 22). Mike, can I see stgt code and some design description, please? You can send it directly on my e-mail address, if necessary.
Vlad
Quoted 21 lines · Vladislav Bolkhovitin
Quoted 19 lines · Dave C Boutcher
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher
This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (
drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.
Quoted 5 lines
Quoted 2 lines
Vlad, can you comment on the state of stgt and whether you see it being ready for mainline any time soon?
Sorry, I can see on stgt page only mail lists archive and not from start (from Aug 22). Mike, can I see stgt code and some design description, please? You can send it directly on my e-mail address, if necessary.
goto the svn page for the code
http://developer.berlios.de/svn/?group_id=4492
As for design desc, I do not have anything. It is the evolving source :) We are slowly merging leasons we learned from open-iscsi, your SCST code, the available software and HW targets, and the SCSI ULD's scatterlist code which needs redoing so it is a bit of a mess.
Re: [RFC] SCSI target for IBM Power5 LPAR/SCST 0.9.3-pre1 published
Quoted 7 lines · Mike Christie
Quoted 1 line · Vladislav Bolkhovitin
Sorry, I can see on stgt page only mail lists archive and not from start (from Aug 22). Mike, can I see stgt code and some design description, please? You can send it directly on my e-mail address, if necessary.
goto the svn page for the code
http://developer.berlios.de/svn/?group_id=4492As for design desc, I do not have anything. It is the evolving source :) We are slowly merging leasons we learned from open-iscsi, your SCST code, the available software and HW targets, and the SCSI ULD's scatterlist code which needs redoing so it is a bit of a mess.
OK, thanks, will try tomorrow.
I put SCST 0.9.3-pre1 on its page (http://sourceforge.net/projects/scst/). This is not the latest, but this is the one, which working. At the end of this week I'll try to put there the latest one as well. Hope, you will learn some more lessons from it :).
Any comments are welcome.
Vlad