Skip to contentSkip to conversation

SCSI target for IBM Power5 LPAR

RFC In review Dave C Boutcher 8 participants 13 messages Sep 6, 2005 – Sep 13, 2005

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 SCSI target for IBM Power5 LPAR

This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.

One of the biggest discussions, I expect, will be whether much of this belongs in user-land. We have had considerable debates on that subject, and ended up with it all in the kernel. Unlike some other targets (e.g. iSCSI) that sit on top of nice TCP/IP stacks, this one interacts at a pretty low level with the firmware. There weren't any good user-land splits that seemed to make sense (and looking at the history of the NFS server also influenced the discussion.)

This is currently just an RFC...if my boxers survive the flamage I'll submit it for inclusion.

Thanks

Dave B

drivers/scsi/ibmvscsi/ibmvscsi.h +1 -0
diff -uNr linux-2.6.13-rc7/drivers/scsi/ibmvscsi/ibmvscsi.h linux-2.6.13-rc7-ibmvscsis/drivers/scsi/ibmvscsi/ibmvscsi.h
@@ -33,6 +33,7 @@
 #include <linux/list.h>
 #include <linux/completion.h>
 #include <linux/interrupt.h>
+#include <scsi/scsi_cmnd.h>
 #include "viosrp.h"
 
 struct scsi_cmnd;
drivers/scsi/ibmvscsi/ibmvscsis.c +3775 -0

2,993 lines not on this page · raw message

drivers/scsi/ibmvscsi/Makefile +2 -0
diff -uNr linux-2.6.13-rc7/drivers/scsi/ibmvscsi/Makefile linux-2.6.13-rc7-ibmvscsis/drivers/scsi/ibmvscsi/Makefile

Lines of this file are not shown. View the raw message.

drivers/scsi/ibmvscsi/srp.h +8 -1
diff -uNr linux-2.6.13-rc7/drivers/scsi/ibmvscsi/srp.h linux-2.6.13-rc7-ibmvscsis/drivers/scsi/ibmvscsi/srp.h

Lines of this file are not shown. View the raw message.

drivers/scsi/Kconfig +10 -0
diff -uNr linux-2.6.13-rc7/drivers/scsi/Kconfig linux-2.6.13-rc7-ibmvscsis/drivers/scsi/Kconfig

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#1 lore raw

Hi Dave,

Quoted 4 lines

This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.

Good stuff. Got a couple of small suggestions.

+/* Allocate a buffer with a dma_address. Don't use dma_alloc_coherent
+ * since that uses GFP_ATOMIC internally and we can tollerate a delay
+ */
+static void *alloc_coherent_buffer(struct server_adapter *adapter, size_t size,
+ dma_addr_t *dma_handle)
+{
+ void *buffer = kmalloc(size, GFP_KERNEL);
+
+ if (buffer) {
+ *dma_handle = dma_map_single(adapter->dev, buffer, size,
+ DMA_BIDIRECTIONAL);
+
+ if (dma_mapping_error(*dma_handle)) {
+ kfree(buffer);
+ buffer = NULL;
+ }
+ }
+
+ return buffer;
+}

This should be fixed in mainline, on ppc64 we no longer build the dma_* ops on top of the pci_* ops. This means we actually look at the flags :)

+ adapter->max_sectors = MAX_SECTORS;

Does this mean we are limited to 128kB transfers? Would it be OK to bump the default?

Anton

#2 lore raw

Hi Anton,

Quoted 4 lines
drivers/scsi/ibmvscsi/ibmvscsis.c +1 -0
+	adapter->max_sectors = MAX_SECTORS;

Does this mean we are limited to 128kB transfers? Would it be OK to bump the default?

We use MAX_SECTORS (which is actually 127.5kB) because that's the max_sectors of the loopback device (we have a lot of users that like the flexibility of using loopback with the ibmvscsis driver)... It can be bumped up without any problems because there is code that splits requests if they are larger than the target's max_sectors...

What would you recommend? 256kB?

Signature · 3 lines
Santiago A. Leon
Power Linux Development
IBM Linux Technology Center
#3 lore raw

Hi,

Quoted 7 lines

We use MAX_SECTORS (which is actually 127.5kB) because that's the max_sectors of the loopback device (we have a lot of users that like the flexibility of using loopback with the ibmvscsis driver)... It can be bumped up without any problems because there is code that splits requests if they are larger than the target's max_sectors...

What would you recommend? 256kB?

~256kB sounds reasonable to me and I notice that other VIO server is setting it to 256kB :)

Anton

#5 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher

This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.

Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.

There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.

Vlad, can you comment on the state of stgt and whether you see it being ready for mainline any time soon?

Signature · 1 line
Dave Boutcher
#6 lore raw

From: Dave C Boutcher <[email protected]>
Subject: Re: [RFC] SCSI target for IBM Power5 LPAR
Date: Wed, 7 Sep 2005 07:45:04 -0500

Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Dave C Boutcher
Quoted 4 lines

This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.

Quoted 6 lines
Quoted 2 lines

Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.

There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.

The generic scsi target framework (stgt) is not SCST (that Vlad has maintained). It is the project that Mike Christie and I started last month. We discussed it with Christoph and decided that it would be better to start from scratch because of the design differences.

#7 lore raw
Quoted 2 lines · FUJITA Tomonori

month. We discussed it with Christoph and decided that it would be better to start from scratch because of the design differences.

Some of the things we are trying to improve upon are things that are better supported in 2.6. Some differences:

- We will support controller hotplug.
- We allow any type of device (dm, scsi, ide, LVM@ etc) as storage. And we do not want hook into SCSI-ml's upper layer drivers and deal with that refcounting if we can help it so we push a lot of code to userspace and only do reads and writes in the kernel.
- We also hope to support any block layer target.
- As mentioned before, scatterlists by using the block layer's support.

There may be more that I am forgetting, but originally we started out by trying to clean up the SCST code for 2.6 and make it resemble SCSI-ml's hotplug model. As we did this it looked like some code could live in userspace and it would end up being a rewrite becuase there was so much to do so we started a new project. We hope to work with Vlad and the SCST developers.

#8 lore raw

Re: remove-single-device removes mounted HDDs (kernel 2.6)

Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Harald Seipp

Christoph Hellwig wrote on 11.08.2005 18:40:03:

Quoted 5 lines · Harald Seipp
Quoted 3 lines

Thank you for your quick responses! It finally looks like the task I need to do - fully rescan all hosts for newly attached/changed devices - must include the logic to skip devices with mounted fs for 2.6.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Why do you want to rescan a full bus anyway? What SCSI transport do you want this for? All transports where new targets can be hot-plugged should be doing automatic scanning already.

Old-fashioned parallel SCSI bus and FC/SAN where there are no hot-plug events AFAIK.

In FC you get per-target events, although none for new LUs showing up on a given target (in fact no SAM transport does the latter)

I believe some Fibre Channel storage arrays generate RSCN's when new logical units are created (although I don't know if the existing Linux Fibre Channel HBA drivers propagate these registered state change notifications to the hot-plug subsystem).

I understand that many iSCSI storage arrays will generate a SCSI Asynchronous Event Notification for the REPORTED LUNS DATA HAS CHANGED Unit Attention condition that SPC-3 says shall be generated when a target changes its logical unit inventory, provided that AEN's are enabled in the control mode page.

So there are SAM transports that generate events when a new LU shows up.

I'm curious as to what plans, if any, exist for handling these events in Linux.

Regards,
-Steve
--------
Steve Byan <[email protected]>
Software Architect
Egenera, Inc.
165 Forest Street
Marlboro, MA 01752
(508) 858-3125

#9 lore raw
In reply to Dave C Boutcher
Quoted 9 lines · Dave C Boutcher
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher

This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.

Quoted 8 lines
Quoted 2 lines

Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.

There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.

stgt is not scst. It is new. And yes we will have proper support for sccatterlists. We need something similar to what sg and st need so I plan on using the block layer functions that are implemented for those upper layer drivers for stgt. If you want to help out and implement that then that would be good. We are also very early in our design so all comments are welcome.

#10 lore raw
In reply to Dave C Boutcher
Quoted 13 lines · Dave C Boutcher
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher

This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.

Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.

Quoted 6 lines

There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.

Vlad, can you comment on the state of stgt and whether you see it being ready for mainline any time soon?

Sorry, I can see on stgt page only mail lists archive and not from start (from Aug 22). Mike, can I see stgt code and some design description, please? You can send it directly on my e-mail address, if necessary.

Vlad

#11 lore raw
Quoted 21 lines · Vladislav Bolkhovitin
Quoted 19 lines · Dave C Boutcher
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Dave C Boutcher

This device driver provides the SCSI target side of the "virtual SCSI" on IBM Power5 systems. The initiator side has been in mainline for a while now (drivers/scsi/ibmvscsi/ibmvscsi.c.) Targets already exist for AIX and OS/400.

Please try to integrate that with the generic scsi target framework at
http://developer.berlios.de/projects/stgt/.

There hasn't been a lot of forward progress on stgt in over a year, and there were some issues (lack of scatterlist support, synchronous and serial command execution) that were an issue when last I looked.

Quoted 5 lines
Quoted 2 lines

Vlad, can you comment on the state of stgt and whether you see it being ready for mainline any time soon?

Sorry, I can see on stgt page only mail lists archive and not from start (from Aug 22). Mike, can I see stgt code and some design description, please? You can send it directly on my e-mail address, if necessary.

goto the svn page for the code
http://developer.berlios.de/svn/?group_id=4492

As for design desc, I do not have anything. It is the evolving source :) We are slowly merging leasons we learned from open-iscsi, your SCST code, the available software and HW targets, and the SCSI ULD's scatterlist code which needs redoing so it is a bit of a mess.

#12 lore raw

Re: [RFC] SCSI target for IBM Power5 LPAR/SCST 0.9.3-pre1 published

Quoted 7 lines · Mike Christie
Quoted 1 line · Vladislav Bolkhovitin

Sorry, I can see on stgt page only mail lists archive and not from start (from Aug 22). Mike, can I see stgt code and some design description, please? You can send it directly on my e-mail address, if necessary.

goto the svn page for the code
http://developer.berlios.de/svn/?group_id=4492

As for design desc, I do not have anything. It is the evolving source :) We are slowly merging leasons we learned from open-iscsi, your SCST code, the available software and HW targets, and the SCSI ULD's scatterlist code which needs redoing so it is a bit of a mess.

OK, thanks, will try tomorrow.

I put SCST 0.9.3-pre1 on its page (http://sourceforge.net/projects/scst/). This is not the latest, but this is the one, which working. At the end of this week I'll try to put there the latest one as well. Hope, you will learn some more lessons from it :).

Any comments are welcome.

Vlad