sg driver and Fedora Core 2

This is slightly off topic (discussing a particular distribution).

The recently released Fedora Core 2 distribution contains a patch that allocates sg device names (e.g. /dev/sg0) only to those SCSI devices _not_ "claimed" by other upper level drivers (i.e. sd, sr, st and osst). There is no kernel (or module) load time parameter to alter this behaviour. There is no way to bind a sg device name to a SCSI device after it has been attached by the SCSI mid level (but perhaps there should be ...).

It seems that the intention of this change is to force cdrecord users to use the /dev/scd or /dev/hd device names (rather than the /dev/sg devices names that were used in the lk 2.0, 2.2 and 2.4 series). While I agree that encouraging the use of the more natural /dev/scd or /dev/hd devices make sense for cdrecord, there are some lesser used applications that are broken by the "forcing" nature of this change (e.g. mtx).

Recently I have had a query about how a specialist application (that worked in the lk 2.4 series) sends 16 MB data through a single SCSI command in Fedora Core 2. The simple answer to that one is the block layer imposes a 128 KB limit on single transfers and that's it. [Counter-intuitively that limit also applies to st and osst when they use the SG_IO ioctl.] Around a year ago I tried to move Jens Axboe on this issue and failed. Evidentally there are good reasons why the block layer imposes that limit. There are other issues with this change.

There seems to be mixed signals coming from the Fedora camp on this change. A "policy" change was one response and this url ( http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123876 ) has Alan Cox stating that this change is a bug.

I am not aware of any such change (or pending change) in the "mainline" linux kernel (i.e. the lk 2.6 source tree controlled by Linus Torvalds).

Doug Gilbert

The block layer does not impose any 128KB limitation. In fact there should be no real limitations (*). The block layer honors whatever restrictions that the drivers sets, 128KB is the maximum transfer size for an ATAPI drive for instance.

You could flag queue capability to handle requests > max_sectors like SCSI has always been able to. Then you could change this check

        if (hdr->dxfer_len > (q->max_sectors << 9))
                return -EIO;

to only fail for drivers that can't partially complete requests correctly. This would allow such applications to continue working.

(*) Not quite true, since the max size of a bio is 1MB on 4KB page sized
hardware. You can string them for bigger total size, though.

That's not the identical issue, it's due to usb storage setting 120KB as the max size causing sg_set_reserved_size() to fail.

Jens Axboe

old relic that isn't even of interesting to the block layer sg "implementation" (we don't keep memory reserved for requests, it's mapped from the user).

Douglas, can I expect applications to break with this patch? They are allowed to set a reserved size of eg 512KB, even if we'll later reject a 256KB command if it's sent to an ATAPI drive.

===== drivers/block/scsi_ioctl.c 1.42 vs edited =====

@@ -89,8 +89,6 @@
 
 	if (size < 0)
 		return -EINVAL;
-	if (size > (q->max_sectors << 9))
-		return -EINVAL;
 
 	q->sg_reserved_size = size;
 	return 0;

Jens Axboe

This is an error. Linus has also vetoed this change. Consider the Fedora kernel wrong on this point. It isnt however your SG_IO issue I think

users seem to say that mtx works as long as you point it to the right device.... after all the ioctls are the same

THe only big difference seems to be that the generic ioctl does enforce queue limits while sg does "something else" when it an invalid-for-the-device IO gets issued..

I don't necessarily regard this as an error: single attachment of ULDs would be very helpful. However, since the mechanics of implementing single attachment are difficult (sg has to be the default attachment of every unclaimed devices, so you need to sort out issues when an ULD is probed *after* sg is attached etc.), I'm very happy for Fedora to demonstrate the feasibility of its implementation before a patch is proposed to mainline.

James

Its most definitely problematic - it breaks a ton of existing tools, and it also doesn't do what you think. Try this for size

insmod scsidriver
open /dev/sg0 (a hard disk)
insmod sd

Apps should migrate to the new facilities when possible but its definitely not a good idea to make this change

Refuse the insertion or do not attach, I suppose. It would be the app's fault for not managing the interface correctly.

Yes, that's why a change like this appeals.

However, as I said, I'm aware it's problematic which is why I'm not about to make any change like it at this point. But, if fedora can demonstrate that it can be made to work sanely and easily, that would be a very powerful argument for inclusion.

James

most of the existing tools take a device as argument, it doesn't (shouldn't) matter if that device name is /dev/sg0 or /dev/sda. After all the ioctls are the same between sg and SG_IO.

So you stop users being able to hotplug controllers while doing firmware updates. Thats not good either

Fedora bugzilla has demonstrated it *doesn't* work. There is a long list of broken third party apps such as drive firmware tools many of which will never include source code. The whole point of sg is arbitary management of devices. Its like /dev/mem and so forth, its power is in the fact it can always be used. This is why Linus vetoed the entire concept.

Fedora hasn't solved the open /dev/sg , insmod case either.

yes cdrecord broke due to a bug in cdrecord and a somewhat discutable behavior difference between SG_IO and /dev/sg when it comes to oversized IO (which cdrecord issues as a result of it's own bug)

*AND SG_IO IS THAT*. You seem to ignore that consistently. What changed is that you don't need top open some OTHER device to send arbitrary commands to /dev/sda, but you can open /dev/sda directly to send those commands, with the same ioctl. THAT changed. Don't tell me that very long list of third party apps ALL doesn't take a device as argument.... undoubtedly several don't but... "long list"...

Unfortunately my devices randomly change name according to what is loaded

I had to recompile my kernel to flash my scsi drive firmware. It works in the generic kernel but not in 2.6.5-Fedora. That alone says its not an acceptable change for a "stable" kernel tree - which hopefully 2.6.x will become in the next six months.

(If you want to argue about 2.7.x then its somewhat different - if scsi
devices appear in sysfs and the sysfs nodes also support SG_IO then the entire notion of /dev/sg could go away in 2.7)

This question only arises if the fw update tools are using the old interface ... I think an increasing level of pain on that front might cause even closed source vendors to consider shifting to the new interface using sd for the disc firmware update.

If there's enough evidence, open minds can be changed.

I think it's too early to declare this experiment in Fedora a failure. If the Fedora release people ultimately decide that the only way to sort all the problems out is to back out the single attachment patch then fine, we'll declare the experiment a failure and not consider anything like it for mainline. However, while fedora is still trying to get everything working with single attachment, I'm not going to prejudge.

James

You think the drive vendors *care* about Linux flash updating tools. You think they will spend anything but the tiniest amount of funding on it. You think anyone who gets told "the major supplier of this OS has arbitarily broken your software during stable releases for no good reason" is likely to update their firmware.

No. Not a chance. Its also nigh on impossible to use /dev/sd to scan the scsi busses for devices because you end up force loading modules, waiting for USB devices to come on line and other nastiness.

Given Linus has vetoed the change for 2.6 the experiment is dead. Its a "won't happen". Doing things with sysfs and 2.7 I can see, since it would be enumerable, would allow you to walk driver and device paths. The Fedora stuff is really irrelevant, the patch is a one person crusade without consideration of the users

Alan

Really what needs to happen is a generic userland tap attaches to a chrdev, issues an ioctl(2) to attach to a request_queue, and then does read(2)/write(2)/mmap(2) to communicate with the request_queue.

:)

Call it /dev/bsg, or somesuch.

The SG_IO ioctl won't work for all cases, since some block devices may rightly wish to limit (or block) multiple openers on the blkdev itself.
OTOH, the SG_IO ioctl does eliminate the userland app needing to
worry about _any_ target/addressing information, since that is implicit in the dentry pointing to the blkdev's inode.

Jeff

Actually, yes, I think Linux is growing enough market momentum for them to take notice.

depending on what your after, device scanning (without any ULDs, including sg) is simple in 2.6; it's ls /sys/bus/scsi/devices.

There's no patch to veto yet. If you redhat people are taking it out of fedora, then, for 2.6, there probably never will be.

I'm not saying "yes" to it's inclusion, I'm just not prepared to say "no" until I see the patch and it gets debated on the list.

James

Hear, hear!

Jens Axboe

Is there a reason that couldn't be part of the block layer sysfs in 2.7.x

Except that I wonder if it should be in sysfs I agree entirely.

It makes it very hard to scan scsi busses, which is what the flash burner app does, and what xsane does (although sane survives ok because there are no kernel scanner device drivers)

Alan

If we act professionally maybe. But breaking that the way FC2 did wasnt professional

and opening those is probably a 2.7 way to get rid of /dev/sg

That is Arjan's call. I already build Fedora 2 kernels with it removed and I'm far from unique in that. The simple fact its an API change makes it a 2.7 kernel thing.

Well, this touches on something else I'm thinking about: lazy attachment. In multi thousand disc environments, you don't want us wasting resources allocating a queue and reading a partition table for all of them; you really only want to do that for the devices you're actually going to use. Once we have this, I think dual attachment will be a dead issue.

Yes, the mapping issues is the huge problem where users keep shooting themselves in the foot.

James

I'll let others haggle over the device name and location <grin> sysfs or whereever is fine with me. I just care about the chrdev's behavior :)

This is precisely the problem with the only-SG_IO approach as well:

AFAICS, you lose the ability to scan the bus, just as you gain the ability to not have to care about scanning the bus. Getting back on topic, I agree with you that the Fedora change [if it is as you describe] is not a good one. And breakage in a stable kernel version is also quite annoying.

Jeff

That would involve supporting ioctls in sysfs, which is something that's been opposed so far. I do agree that you want to flash disc firmware using a hierarchical path rather than a flat namespace that you have to double-check to be sure you're not flashing your scanner with disc firmware.

Possibly we could have /sys/bus/scsi/devices/1\:0\:1\:0/raw_io that internally uses the SG_IO codepath. We'd need to rewrite the raw file interface to support the large IOs necessary, but that's certainly doable.

"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

That is actually a dying issue (aside from the whole /dev/sg debate). Users select the CD burner by vendor/device name from the pull down menu in real world computing. The underlying question is how the GUI application does the scan and how it figures out what burner ties to what cd-rom device in the file manager. SG_IO is wonderful here and has solved the latter part nicely. sysfs has solved the former part.

Alan

Regardless of lazy attachment, request_queues need to support multiple simultaneous producers of struct request's.

Jeff

That has always been the case, even if it was just driver private use before. As such there is no dual-attachment issue for sg.

Jens Axboe

Jeff + Jens,
Yes, that is what I was getting at with my "bind" idea. Could a "misc" char device minor number for /dev/bsg be made available? It isn't clear to me how sysfs can be used to do what Jeff proposes.

I have no expertise in Fibre Channel but Serial Attached SCSI (SAS) throws up more addressing problems. There can be up to 16,000 devices in a single SAS domain the bulk of which would be targets **, expanders which are the interesting ones, and other initiators (potentially on other machines) will also be visible. Those expanders are critical to configuring the SAS domain. Expanders are SAS devices but not SCSI devices (i.e. they are neither initiators nor targets ***). It seems obvious that the management of a SAS domain should be done from the user space. Only those expanders and devices directly attached to a SAS Host Bus Adapter should automatically be visible.

Enumerating all the disks, and/or all the scsi devices (as either devices or sysfs entries) makes less and less sense. In the case of SAS, devices can be discovered by recursive descent of the expanders (via a bind through /dev/bsg for example). In a large system one possible policy could be: only if the user is interested in a particular device should it be presented as a device node (in /dev, /udev or /sys).

My recent patch to scsi_debug (adding up to 4 partitions per pseudo device) "allows" (6 * 32768) device nodes to be created! To play this game, turn off udev, hotplug and set delay=0 in scsi_debug:-) I didn't get far with 512 MB of ram, Patrick Mansfield got further with a multi gigabyte machine. Seems like sysfs eats up most of the ram. To isolate out the impact of the block subsystem the pseudo devices can be set to be enclosure peripheral type (for example) via a scsi_debug driver argument.

** targets include both SATA and SAS disks

*** show should SAS expanders go in /sys/class/scsi_device ?

Doug Gilbert

Is there any reason that the SAS domain should in itself be a character device file. I've always thought of it as a network and anticipated a network layer interface for such things. That might just be my background however.

Alan,
In the SCSI parallel world the "domain" is usually called the "bus" which is a set of targets and one or more initiators and the ribbon (or external) cable that interconnect them.

In SAS the "domain" is a little more rubbery. After wide links are taken into account, there cannot be multiple paths between scsi devices within one SAS domain (i.e. "no loops"). An expander is part of the service interconnect subsystem of a _single_ SAS domain (i.e. it can't be shared). This means that expanders can be wired incorrectly. Even simple interconnects need some thought:

    |   SAS  2   |
    |  port HBA  |
    --------------
        |     |
        |     |
        |     |
        |     |
     -------------
     | SAS dual  |
     |ported disk|

Is this a (2 phy) wide-link, single SAS domain or two SAS domains? On current indications it is two SAS domains since dual ported SAS disks have different SAS addresses on each port. [The fact that the SAS HBA has the same SAS addresses on both phys only meets half the requirement for a wide link.]

See SAS Quiz question 1 (then its answer) in the pdfs at:
http://www.scsita.org/aboutscsi/sas/tutorials.html
for a more complex scenario.

Detecting and explaining to a user why his/her SAS domain(s) is incorrectly wired will be an interesting challenge.

So it wasn't the SCSI domain that I was proposing should be a char (or network) device file. It was the SMP target port inside each expander that is used for discovery, configuration and statistics that needs a packet based initiator driver/interface (like the sg driver). The concept of a SAS domain I would rather leave for a user space program.

Doug Gilbert