Proposed changes to generic blk tag for use in SCSI (1/3)

The layers can cope with the switch easily enough. The problem is that to send an untagged command to a SCSI device you have to wait for the outstanding tags to clear which is what causes the stall. The scsi mid-layer queue push back system pushes all commands back to the BIO layer marked as REQ_SPECIAL (because the upper layer drivers generate the commands and it has no idea what they are supposed to be doing) if the driver cannot handle them. This means for those drivers (like the new adaptec) which load up the device until it returns a queue full (thus causing push back into the bio layer) we'd get stutter in the command pipeline. The cleanest solution is to allow (but not require) tagging of every request type.

I thought about doing this. The problem is that the blk layer doesn't have very good instrumentation for detecting the condition. The SCSI layer is the one that has per command timers and all the other necessaries so it can detect when a command should have returned and take corrective action.

James

The layers can cope with the switch easily enough. The problem is that to send an untagged command to a SCSI device you have to wait for the outstanding tags to clear which is what causes the stall. The scsi mid-layer queue push back system pushes all commands back to the BIO layer marked as REQ_SPECIAL (because the upper layer drivers generate the commands and it has no idea what they are supposed to be doing) if the driver cannot handle them. This means for those drivers (like the new adaptec) which load up the device until it returns a queue full (thus causing push back into the bio layer) we'd get stutter in the command pipeline. The cleanest solution is to allow (but not require) tagging of every request type.

I thought about doing this. The problem is that the blk layer doesn't have very good instrumentation for detecting the condition. The SCSI layer is the one that has per command timers and all the other necessaries so it can detect when a command should have returned and take corrective action.

James

Well, intentional behaviour is hardly what I would call a stall. I thought you were implying that it would stall the queue indefinitely. I'm fully aware that it forces the queue to wait until all outstanding commands have completed before sending the untagged command, that's part of the desired behaviour in that case.

This I'm not following. If you get a QUEUE_FULL from the adaptec driver, then the commands you are pushing back should still be tagged and no stall should be required beyond just waiting for any outstanding command on the drive to complete or for a timeout to pass. It should not require any untagged type stall where you have to drain the entire pipeline...

I would think that, eventually, the bio layer will support I/O fencing via tagged commands (aka, ext3 needs an I/O fence and the bio layer does as needed to enforce that, which on scsi may mean an ordered queue tag is generated instead of a regular tag and on IDE it may mean something else). It will have to be able to tell that some of these conditions have been satisfied in those cases, so I see no reason why it shouldn't be made aware of them now. Just my $.02

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Ah, but that was the problem in the blk generic tagging. only requests of type REQ_CMD get tagged, so say the SD driver gets a block read request as a REQ_CMD, translates it into a SCSI READ, sends it through the mid-layer to the low layer driver which requests a block layer tag but eventually responds QUEUE_FULL. Now the command gets pushed back to the blk queue head as a REQ_SPECIAL (and as part of the push back, we have to finish the tag since command moves from the tag queue to the blk queue), which to the scsi mid layer means request with already formulated SCSI command so don't go back through the upper layer driver again. The problem is that when this command comes back again into the scsi mid layer for execution it would now do so as an untagged command because the blk_queue_start_tag() code will only tag REQ_CMD requests, and hence we get a queue stall every time the low level driver responds QUEUE_FULL.

This was the behaviour (in the blk layer) I was objecting to---on the second go around, we request a tag using blk_queue_start_tag() but get denied because the request isn't of the correct type---and why I think the blk_generic_start_tag() needs to allow REQ_SPECIAL requests to be tagged.

I've actually already put this code into the mid layer patch (the scsi_populate_tag_msg() function in scsi.h) to generate an ordered tag for the case where the request is marked REQ_BARRIER.

James

They are, but you can drop it (as long as you reacquire it before exit).

This isn't necessarily true. Some SCSI cards have a single mailbox for outgoing and incoming transfer parameters and commands. If you have one of these, you need to keep interrupts disabled as you set up a command so that your own interrupt routine doesn't overwrite the values if you get a reconnect interrupt while you're programming the card.

This is a very important point to bear in mind. Since the io_request_lock must be held when calling the done() routine, you are asking to create a deadlock if you use your own locks in the driver (it's not impossible to get this correct, just very difficult).

Basically, the io_request_lock is used to protect structures within the SCSI and block layer. Once you get into the LLD, you may extend it if you wish to protect the individual data structures of your driver (which is why it's held on entry to most of the LLD API) but if you don't want to, it is safe to drop it (as long as your queuecommand is re-entrant and you re-acquire it before exiting the API routines).

James

Then you better not *ever* release the lock under those conditions. Leaving interrupts disabled may work on UP, but on SMP it only disables interrupts on that one CPU and if you aren't holding your lock then another CPU can run the interrupt routine and trash your data. So, back to my point, if you aren't enabling interrupts, then there is no reason to release the lock because you obviously need some sort of lock type protection.

Don't get too hung up on this point. In 2.5 you don't need either the io_request_lock or host->host_lock when calling the done function. In 2.4.19 you don't need it if you are a new eh driver, only if you are old. And hopefully by 2.4.20 that won't be true either. If Marcello takes the patch I've got for 2.4 that we shipped in our Red Hat 7.3 product, then both new and old eh drivers will no longer have to hold a lock when they call the done function because both of them will be using the new eh code's bottom half mechanism.

Again, if Marcello takes my patch, this will become easier as well since it adds a host->lock lock that allows you to get your driver off the horrible io_request_lock once and for all. (Note: I may have to change the name to host->host_lock to make it match 2.5, but that just annoys the hell out of me because I *detest* totally redundant names like that...)

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Exactly so.

Well, that's just semantics, I won't get into the argument...

Note also that 2.5 already has per queue locks (that means per individual SCSI devices) which are lock pointers. In 2.5 current, all the devices that hang off an individual host have their queue_locks initialised to point to the single host->host_lock. However, drivers which can do it (the type of multi-threaded multi-mailbox ones) will be permitted to use individual and separate queue_locks, rather than the single host_lock, to protect their mailbox registers. If you are never going to do this, the difference between using host_lock and queue_lock is irrelevant to you. If you are, you need to begin using the queue_locks.

James

Such code looks a bit odd, but I would rather have it and be able to more easily find or audit all the references; using cscope, there are about 6700 references to lock in the 2.5 kernel (only looking at i386 sources, so there are probably more), and about 300 references to host_lock.

-- Patrick Mansfield

James,
Maybe I am mis-reading something.

Why would we want to go back to a model of having LL drivers using locks outside there domain (queue_lock). Should we not be moving toward having drivers use there own locks.

I would think a future model might be to have queue_locks for queue operations and possible shared as a scsi device lock, a host_lock for host data, and LL driver locks used where the driver needs them.

-Mike

Michael Anderson
[email protected]

I've assumed that you also wanted to add that semantics are quite _very_ important.

Luben

Unfortunately, it would destroy the REQ_BARRIER approach in the block layer. At best, abort probably causes a command to overtake a barrier it shouldn't, at worst we abort the ordered tag that is the barrier and transactional integrity is lost.

When error correction is needed, we have to return all the commands for that device to the block layer so that ordering and barrier issues can be taken care of in the reissue. This makes LUN RESET (for those that support it) the minimum level of error correction we can apply.

James

What do we plan to do for the cases where reset is disabled because we have shared disk scsi so don't want to reset and hose the reservations ?

If your error correction always requires all commands return to the block layer then the block layer is IMHO broken. Its messy enough doing that before you hit the fun situations where insert scsi commands of their own the block layer never initiated.

Next you only need to return stuff if commands have been issued between the aborting command and a barrier. Since most sane systems will never be causing REQ_BARRIER that should mean the general case for an abort is going to be fine. The CD burner example is also true for this. If we track barrier sequences then we will know the barrier count for the command we are aborting and the top barrier count for commands issued to the device. Finally you only need to go to the large hammer approach when you are dealing with a media changing command (ie WRITE*) - if we abort a read then knowing we don't queue overlapping read then write to disk we already know that the read will not break down the tag ordering as I understand it ?

If we get to the point we need an abort we don't want to issue a reset. Not every device comes back sane from a reset and in some cases we have to issue a whole sequence of commands to get the state of the device back (door locking, power management, ..)

REQ_BARRIER is used for filesystem devices (disks mainly) while the devices that might benefit most from working aborts would likely be other devices. But, regardless, the REQ_BARRIER ordering *can* be preserved while using abort processing. Since the command that needs aborting is, as you are hypothesizing, before the REQ_BARRIER command, and since it hasn't completed, then the REQ_BARRIER command can not be complete and neither can any of the commands behind the REQ_BARRIER. On direct access devices you are only concerned about ordering around the barrier, not ordering of the actual tagged commands, so for abort you can actually call abort on all the commands past the REQ_BARRIER command first, then the REQ_BARRIER command, then the hung command. That would do the job and preserve REQ_BARRIER ordering while still using aborts.

Not really, this would be easily enough done in the ML_QUEUE area of the scsi layer, but it matters not to me. However, if you throw a BDR, then you have cancelled all outstanding commands and (typically) initiated a hard reset of the device which then requires a device settle time. All of this is more drastic and typically takes longer than the individual aborts which are completed in a single connect->disconnect cycle without ever hitting a data phase and without triggering a full device reset and requiring a settle time.

Not true.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

James -

If we only send an abort or reset after a quiesce I don't see why one is better than the other.

Not specific to reset or abort - if a single command gets an error, we wait for oustanding commands to complete before starting up the error handler thread. If all the commands (error one and outstanding) have barriers, those that do not error out will complete out of order from the errored command.

How is this properly handled?

And what happens if one command gets some sort of check condition (like medium error, or aborted command) that causes a retry? Will IO's still be correctly ordered?

The abort could also be usefull in handling the locking/ownership of the scsi_cmnd - the abort at the LLD layer can be used by the LLD to cancel any software timeouts, as well as to flush the command from the hardware. After the abort, the mid-layer could assume that it once again "owned" the scsi_cmnd, especially if the LLD abort were a required function.

I would like to see error handling occur without quiescing the entire adapter before taking any action. Stopping all adapter IO for a timeout can be a bit expensive - imagine a tape drive and multiple disks on an adapter, any IO disk timeout or failure will wait for the tape IO to complete before allowing any other IO, if the tape operation is long or is going to timeout this could be minutes or hours.

-- Patrick Mansfield

The reset gets issued and the reservation gets broken. Good HA or other software knows the reservation may be lost and takes this into account in the cluster algorithm.

With SCSI-2 reservations, there's no way to preserve the reservation and have the reset be effective (I know, in theory, that this can be circumvented by the soft reset alternative, but I've never seen a device that implements it correctly). I suppose we hope SCSI-3 Persistent Group Reservations come along quickly.

This is part of the slim SCSI down approach. The block layer already has handling for tag errors like this. Inserted SCSI commands should now work correctly since we're deprecating the scsi_do_cmnd() in favour of scsi_do_req, which means the command is always associated with a request and goes into the block queue just like any other request.

I think the block layer, which already knows about the barrier ordering, is the appropriate place for this. If you think the scsi error handler is a hairy wart now, just watch it grow into a stonking great carbuncle as I try to introduce it to the concept of command queue ordering and appropriate recovery.

I agree with your reasoning. However, errors occur infrequently enough (I hope) so that its just not worth the extra code complexity to make the error handler look for that case.

However, in all honesty, I have to say that I just don't believe ABORTs are ever particularly effective. As part of error recovery, If a device is tipping over into failure, adding another message isn't a good way to pull it back. ABORT is really part of the I/O cancellation API, and, like all cancellation implementations, it's potentially full of holes. The only uses it might have---like oops I didn't mean to fixate that CD, give it back to me now---aren't clearly defined in the SPEC to produce the desired effect (stop the fixation so the drive door can be opened).

Well, this is SCSI---the first thing most controllers do for parallel SCSI at least is reset the BUS. Some FC drivers do the FC equivalent as well (not that they should, but that's another issue).

The pain of coming back from a reset (and I grant, it isn't trivial) is well known and well implemented in SCSI. It also, from error handlings point of view, sets the device back to a known point in the state model.

James

You are correct. However, as soon as you abort the problem command (assuming the device recovers from this), it will go on its merry way processing the remaining commands in the queue. Assuming one of these is the barrier, you've no way now of re-queueing the aborted command so that it comes before the ordered tag barrier. You can try using a head of queue tag, but it's still a nasty race.

I agree, but the most likely scenario is that now you're trying to abort almost every tag for that device in the system. Isn't reset a simpler alternative to this?

I agree. I certainly could do it. I'm just a lazy so-and-so. However, think what it does. Apart from me having to do more work, the code becomes longer and the error recovery path more convoluted and difficult to follow. The benefit? well, error recovery might be faster in certain circumstances. I just don't see that it's a cost effective change. If you're hitting error recovery so often that whether it recovers in half a second or several seconds makes a difference, I'd say there's something else wrong.

James

Point taken

When you ar ehandling CD-ROM problems then they can be quite a lot. Not helped by the fact some ancient CD's seem to like to take a long walk in the park when told to reset.

Yes but does windows assume it. There are two specs 8)

Resetting the bus is antisocial to say the least. We had lots of hangs with the old eh handler that went away when the new one didnt keep resetting the bus. If we do reset the bus then we have to handle "Sorry can't reset the bus" (we have cards that can't or won't) as well as being conservative about timings, making sure we don't reset the bus during the seconds while a device rattles back into online state and so on.

Quiesce means from the top (no more commands go from the block layer to the device) not from the bottom (commands can still be completed for the device).

We don't wait. The commands may possibly be completing in parallel with recovery. To address your point, though, that's why the device needs to be reset as fast as possible: to preserve what's left of the command order. I accept that for a misbehaving device, this may be a race.

Retries get eliminated. It should be up to the upper layers (sd or beyond) to say whether a retry is done. Since, as you point out, retries automatically violate any barrier, it is probably up to the block layer to judge what should be done about the problem.

Actually, I do think that quiescing has an important role to play. A lot of drive errors can be self inflicted indigestion by accepting more tags into the queue than it can process. Quiescing the queue lets us see if the drive can digest what it currently has or whether we need to apply a strong emetic.

I'd actually like to add tag starvation recovery to the error handler's list of things to do. In that case, all you do on entry to the error handler is quiesce the upper queue and wait a while to see if the commands continue returning. You only begin more drastic measures if nothing comes back in the designated time period.

James

(Solution to this race was in my next paragraph as you found ;-)

Not really. It hasn't been done yet, but one of my goals is to change the scsi commands over to reasonable list usage (finally) so that we can avoid all these horrible linear scans it does now looking for an available command (it also means things like SCSI_OWNER_MID_LAYER can go away because ownership is defined implicitly by list membership). So, basically, you have a list item struct on each command. When you build the commands, you add them to SDpnt->free_list. When you need a command, instead of searching for a free one, you just grab the head of SDpnt->free_list and use it. Once you've built the command and are ready to hand it off to the lldd, you put the command on the tail of the SDpnt->active_list. When a command completes, you list_remove() it from the SDpnt->active_list and put it on the SDpnt->complete_list to be handled by the tasklet. When the tasklet actually completes the command, it frees the scsi command struct by simply putting it back on the SDpnt->free_list. Now, if you do things that way, your reset vs. abort code is actually pretty trivial.

Case 1: you want to throw a BDR. Sample code might end up looking like this,

	[ oops we timed out ]
	hostt->bus_device_reset(cmd);
	if(!list_empty(cmd->device->active_list)) {
		[ our commands haven't all been returned, spew chunks! ]
	}
	[ do post reset processing ]

Case 2: you want to do an abort, but you need to preserve ordering around any possible REQ_BARRIERs on the bus. This requires that we keep a REQ_BARRIER count for the device, it is after all possible that we could have multiple barriers active at once, so as each command is put on the active_list, if it is a barrier, then we increment SDpnt->barrier_count and as we complete commands (at the interrupt context completion, not the final completion) if it is a barrier command we decrement the count.

	[ oops we timed out ]
	while(SDpnt->barrier_count && cmd) {
		// when the aborted command is returned via the done()
		// it will remove it from the active_list, so don't remove
		// it here
		abort_cmd = list_get_tail(SDpnt->active_list);
		if(hostt->abort(abort_cmd) != SUCCESS) {
			[ oops, go on to more drastic action ]
		} else {
			if(abort_cmd->type == BARRIER)
				SDpnt->barrier_count--;
			if(abort_cmd == cmd)
				cmd = NULL;
		}
	}
	if(cmd) {
		if(hostt->abort(cmd) != SUCCESS)
			[ oops, go on to more drastic action ]
	}

Now, granted, that is more complex than going straight to a BDR, but I have to argue that it *isn't* that complex. It certainly isn't the nightmare you make it sound like ;-)

Well, as I've laid it out above, I don't really think it's all that much to implement ;-) At least not in the mid layer. The low level device drivers are doing *far* more work to support aborts than the mid layer has to do.

Matter of some question, I'm sure. I don't see it as all that much work, so it seems reasonably cost effective to me ;-)

Hehehe, if you are hitting error recovery at all then something else is wrong by definition, the only difference is in how you handle it :-P

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Hmmm...I thought a big reason for adding REQ_BARRIER was to be able to support more robust journaling with order requirement verification. If that's true, then REQ_BARRIER commands could become quite common on disks using ext3.

                             ^^^^^^^^^^^^^^^^^^^^^^

Then you might as well skip device resets since they are implemented using messages and go straight to bus resets. Shot deflected, no score.

In my experience, aborts have always actually worked fairly well in any scenario where a bus device reset will work. Generally speaking, the problems I've always ran into with SCSI busses have been either A) this command is screwing up but it isn't confusing the drive so we can abort it or BDR it because the drive still responds to us or B) the bus is hung hard and no transfers or messages of any kind can make it through. In the B case, a full bus reset is the only thing that works. In the A case, aborts work just as often as anything else.

So does a successful abort.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Oops, delete those last two lines....the done() function decrements the barrier_count for us.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Then we need to give the block layer a lot more information about what kind of a problem occurred

I doubt it very much because the only way to implement barriers on IDE is fantastically expensive. I'm dubious it makes sense for ext3 to use barriers in that way. All it needs as I understand the rules is that an I/O isnt reported as completed by the device driver before it is on the medium or in a non volatile cache.

At least partly done in 2.4-ac and waiting pushing to Marcelo. I believe Hugh contributed the O(1) command finder

Hmm. There do seem to be a lot of loopholes/race conditions where the barrier just won't work right in the face of error recovery. I wouldn't want to use barriers on any system where data integrity was crucial.

For example, in Fibrechannel using class 3 (the usual)

send command (command frame corrupted; device does not receive) send barrier (completes normally)
... (lots of time goes by, many more commands are processed) timeout original command whose command frame was corrupted

The only safe way to run such a filesystem is to hold the barriers in the driver until all previous commands are successfully completed.

There was also the problem of the queue full to the barrier command, etc.

Did I miss the answer to these? I don't recall seeing an answer to Patrick Mansfield's questions either (original message edited cutting out a couple of paragraphs):

jeremy

Not that I want to contradict, but an example.
Without my sd.c patch from yesterday or so (fixing MODE SENSE calls) an "insmod usb-storage.o" would take 14 minutes and 6 seconds for me.

[One USB device, with 3 subdevices, gets into a bad state when presented with a MODE SENSE command that asks for more than the 56 bytes it has available. For each of the three subdevices we get a long sequence of retries, abort, reset, host reset, bus reset before it is taken off-line.]

The scsi error recovery has many bad properties, but one is its slowness. Once it gets triggered on a machine with SCSI disks it is common to have a dead system for several minutes. I have not yet met a situation in which rebooting was not preferable above scsi error recovery, especially since the attempt to recover often fails.

Andries

This does not have to be this way. It is a solvable problem.

Yes, well, this too is solvable. It, in fact, reminds me that one of the things I think needs added to the scsi host settings is a default timeout value for typical devices. Something like adding a default_timeout value to each Scsi_Device struct and allowing the scsi driver to modify the value during the slave_attach() call. Then we can put the default timeout on non-intelligent controllers to something sane while things like MegaRAID controllers can keep their sky high timeout values.

This, too, is solvable. It just requires that the scsi subsystem start paying attention to *how* things fail and making the error handling code smart enough to know when to retry things and when to just fail.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Well I for one prefer the scsi timeout/abort sequence on a CD getting confused badly by a bad block (as at least some of my drives do) to a reboot everytime I get bad media

It's three times longer even in pseudocode...

However, assume we do this (because we must for barrier preservation). The chances are that for a failing device we're aborting a significant number of the tags. This is quite a big increase in the message load over what we do now---Particularly for the AIC driver which can have hundreds of tags outstanding (murphys law says it's usually the earilest tag which times out). I'm not convinced that a BDR, which is a single message and has roughly the same effect, isn't preferable.

However, what about a compromise? We can count outstanding commands, so what about doing abort *if* the number of outstanding commands is exactly one (the one we're trying to abort). This means for devices that don't do TCQ (like old CD-ROMs) we'll always try abort first. For large numbers of outstanding tags, we skip over abort and move straight to BDR. The code to implement this will be clean and simple because abort no longer has to pay attention to the barrier.

James

This doesn't look right to me from the SCSI angle I don't see how you can get a successful disconnect on a command the device doesn't receive (I take it this is some type of Fibre magic?). Of course, if the device (or its proxy) does receive the command then the ordered queue tag implementation requires that the corrupted frame command be processed prior to the barrier, this isn't optional if you obey the spec. Thus, assuming the processor does no integrity checking of the command until it does processing (this should be a big if), then we still must get notification of the failed command before the barrier tag is begun. Obviously, from that notification we do then race to eliminate the overtaking tags.

The queue full problem still exists. I've used this argument against the filesystem people many times at the various fora where it has been discussed. The situation is that everyone agrees that it's a theoretical problem, but no-one is convinced that it will actually occur in practice (I think it falls into the "risk we're willing to take" category).

James

Reboot is bad. Retries are bad.

Errors should be returned to an upper layer, with an error code: "may retry", or "will never work". (Like in SMTP)

I will most likely set the "retry count" to 0: Never retry. Almost never works anyway. And the disk already retried manytimes, so retrying in software is only "taking time".

We do datarecovery around here. We get bad disks on a dayly basis. We are currently reading a drive that gets over 10Mb per second while spitting out bad block reports!

Thing is: those blocks that didn't work first time, may work on our second retry. However, we need userspace control over that retry. We prefer to get the 18G worth of data off the disk first, and only then retry the blocks that happened to be bad first time around.

Roger.

** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currenly in such an       * 
* excursion: The stable situation does not include humans. ***************

To make this work, you really need to use the QErr bit in the disconnect/reconnect page and/or ECA or ACA. QErr I believe is well supported in devices, but ECA (pre SCSI-3) and ACA most likely receive very little testing.

I will also voice my opinion (again) that watchdog timer recovery is in the wrong place in Linux. It belongs in the controller drivers:

1) Only the controller driver knows when to start the timeout
2) Only the controller driver knows the current status of the bus/transport
3) Only the controller can close timeout/just completed races
4) Only the controller driver knows the true transport type (SPI/FC/ATA/USB/1394/IP) and what recovery actions are appropriate for that transport type given the capabilities of the controller.
5) The algorithm for recovery and maintaining order becomes quite simple:
1) Freeze the input queue for the controller
2) Return all transactions unqueued to a device to the mid-layer
3) Perform the recovery actions required
4) Unfreeze the controller's queue
5) Device type driver (sd, cd, tape, etc) decides what errors at what rates should cause the failure of a device. The controller driver just needs to have the error codes so it can honestly and fully report to the type driver what really happens so it can make good decissions

This of course assumes that all transactions have a serial number and that requeuing transactions orders them by serial number. With QErr set, the device closes the rest if the barrier race for you, but even without barriers, full transaction ordering is required if you don't want a read to inadvertantly pass a write to the same location during recovery.

For prior art, take a look at FreeBSD. In the worst case, where escalation to a bus reset is required, recovery takes 5 seconds.

--
Justin

In FC class 3 if you are logged into a port then notice of this loss doesn't happen until a upper level timeout occurs (ULTP?). The loss can happen prior to the command reaching the device (i.e. the switch can drop the frame). If a corrupted frame makes it to the device it will be discarded as there is not much it can do with a frame containing unreliable data. In FC class 2 frames are ack'd so the recovery can be much more responsive.

-Mike

Michael Anderson
[email protected]

Hi,

fd = open(pathname, O_NORETRY); ?

Thunder

--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-

Using the struct list_head for this will literally allow you to do _magic_. Avoiding the linear scan is the last thing this will fix.

It would allow for a lot better/simpler/sound design of all of the mid layer/SCSI core. Things will be/become easier as you point out below. Currently the mid-layer queuing is hairy at best.

I'm all for it.

Great!

Once you're on that train, you may want to rethink the whole queuing mechanism of the mid-layer (straight from sd/etc and internally down to LLDD) for an improved design.

There'd be problems like cmd moving b/n lists is atomic, only cmd movers can actually cancel a command, move before calling queuecommand(), etc, but is nothing extraordinary.

No, it certainly is NOT!

Granted, by looking at the code it will not be overly clear who moves what and when, but a 2 page commentary on the design would only leave one exlaiming ``Aaaaah... such simplicity, so great!''

Right, it's not. This type of queuing mechanism would only make things more consistent and easy to manipulate.

There'd be logistical issues, but those are easy to figure out with pen and paper.

Luben

``Perfection is achieved not when there is nothing more to add
  but when there is nothing left to take away.''
                              Antoine de Saint Exupery

The original FCP (SCSI commands over Fibre) profile specified that QERR=1 was not available. Unless that is changed, it would appear that you cannot count on being able to set Qerr.

Qerr was one of those annoying little things in SCSI that forces host adapter drivers to know a mode page setting.

jeremy

You don't have disconnect per se with Fibre (or with other packet based interfaces). A command is sent as a frame. If the frame is corrupted or lost on the way to the target, then the target will never know about the command having been sent. There are obvious problems if that command has an ordered tag, preceded and followed by simple tags dependent on the ordered semantics being followed, and where the simple tagged commands following don't wait for the ordered tag to finish. There are other failure modes too, of course.

I guess it all depends on how sensitive one is to getting it wrong. In any case, it looks as though you are doing what you can . . . .

thanks

jeremy

It is not the controllers that know, but the type drivers. The controller drivers should be conduits for commands, nothing more. With the proper events and error codes, the type drivers can maintain mode parameters and only the type drivers know what parameters are appropriate for their type of service.

For FC, you need to use ECA/ACA anyway as that is the only way to deal with inflight commands at the time of an error.

--
Justin

But don't the controllers have to know which commands have been silently aborted by the target? There are various resources allocated to a command, so if the command will never be completed, as a result of some other command getting "check condition", the controller would have to know.

Or are you suggesting that the type drivers would tell the controllers to release those resources?

And would this change if the controller drivers were keeping track of timeouts, etc.?

jeremy

The controller drivers would need to know that the QErr policy is in effect for a given device. With that knowledge, the book keeping from the controller's standpoint is really quite simple and no different than if a lun reset, target reset, clear task set, bus reset or any other task management function is executed. In other words, the controller drivers already need to understand the consequences of such events.

The controller knows which commands are affected and just returns them to the mid-layer with the appropriate error code.

Not necessary.

This all assumes that the controllers are doing watchdog recovery.

--
Justin

Right. We're on the same page now. So you'd need some interface for the sd driver (or midlayer) to tell the host driver what the QErr policy is for a given lun. I don't recall seeing that in the current layer, but it would be easy to add, presumably.

jeremY

This patch makes it possible to adjust the queue depth of a scsi device after it has been in use some time and you have a better idea of what the optimal queue depth should be. For the most part this should work, but my 2.5.40 test machine is blowing chunks on the serverworks IDE support right now so it isn't tested :-(

What I need, people to test this with the old aic7xxx driver (which implements the new code paths, no other drivers do) and people to test with other drivers to make sure it doesn't break them. If I don't hear complaints then I'll move on to my next change which is far more intrusive on drivers in general.

Side note: I left the control of queue depth setting solely in the hands of the low level drivers since they are the *only* ones that can get an accurate queue depth reading at the time of any given QUEUE_FULL message (see what the aic7xxx_old driver has to do in the QUEUE_FULL handler to find out how many commands the drive has seen at this exact point in time vs. how many we may have queued up to the card, the difference in numbers can be significant). For that reason, the adjust_queue_depth call was made to defer the action until later so that it was interrupt context safe.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

@@ -46,7 +46,9 @@
 	eh_host_reset_handler: NULL,				\
 	abort: aic7xxx_abort,					\
 	reset: aic7xxx_reset,					\
-	slave_attach: NULL,					\
+	select_queue_depths: NULL,				\
+	slave_attach: aic7xxx_slave_attach,			\
+	slave_detach: aic7xxx_slave_detach,			\
 	bios_param: aic7xxx_biosparam,				\
 	can_queue: 255,		/* max simultaneous cmds      */\
 	this_id: -1,		/* scsi id of host adapter    */\

@@ -64,6 +66,8 @@
 extern int aic7xxx_reset(Scsi_Cmnd *, unsigned int);
 extern int aic7xxx_abort(Scsi_Cmnd *);
 extern int aic7xxx_release(struct Scsi_Host *);
+extern int aic7xxx_slave_attach(Scsi_Device *);
+extern void aic7xxx_slave_detach(Scsi_Device *);
 
 extern const char *aic7xxx_info(struct Scsi_Host *);
 

@@ -481,6 +481,7 @@
 extern void scsi_bottom_half_handler(void);
 extern void scsi_release_commandblocks(Scsi_Device * SDpnt);
 extern void scsi_build_commandblocks(Scsi_Device * SDpnt);
+extern void scsi_adjust_queue_depth(Scsi_Device *, int, int);
 extern void scsi_done(Scsi_Cmnd * SCpnt);
 extern void scsi_finish_command(Scsi_Cmnd *);
 extern int scsi_retry_command(Scsi_Cmnd *);

@@ -563,6 +564,8 @@
 	volatile unsigned short device_busy;	/* commands actually active on low-level */
 	Scsi_Cmnd *device_queue;	/* queue of SCSI Command structures */
         Scsi_Cmnd *current_cmnd;	/* currently active command */
+	unsigned short queue_depth;	/* How deep of a queue we have */
+	unsigned short new_queue_depth; /* How deep of a queue we want */
 
 	unsigned int id, lun, channel;
 

@@ -586,24 +589,25 @@
 	unsigned char current_tag;	/* current tag */
 	unsigned char sync_min_period;	/* Not less than this period */
 	unsigned char sync_max_offset;	/* Not greater than this offset */
-	unsigned char queue_depth;	/* How deep a queue to use */
 
 	unsigned online:1;
 	unsigned writeable:1;
 	unsigned removable:1;
 	unsigned random:1;
-	unsigned has_cmdblocks:1;
 	unsigned changed:1;	/* Data invalid due to media change */
 	unsigned busy:1;	/* Used to prevent races */
 	unsigned lockable:1;	/* Able to prevent media removal */
 	unsigned borken:1;	/* Tell the Seagate driver to be 
 				 * painfully slow on this device */
-	unsigned tagged_supported:1;	/* Supports SCSI-II tagged queuing */
-	unsigned tagged_queue:1;	/* SCSI-II tagged queuing enabled */
 	unsigned disconnect:1;	/* can disconnect */
 	unsigned soft_reset:1;	/* Uses soft reset option */
-	unsigned sync:1;	/* Negotiate for sync transfers */
-	unsigned wide:1;	/* Negotiate for WIDE transfers */
+	unsigned sdtr:1;	/* Device supports SDTR messages */
+	unsigned wdtr:1;	/* Device supports WDTR messages */
+	unsigned ppr:1;		/* Device supports PPR messages */
+	unsigned tagged_supported:1;	/* Supports SCSI-II tagged queuing */
+	unsigned tagged_queue:1;	/* SCSI-II tagged queuing enabled */
+	unsigned simple_tags:1;	/* Device supports simple queue tag messages */
+	unsigned ordered_tags:1;/* Device supports ordered queue tag messages */
 	unsigned single_lun:1;	/* Indicates we should only allow I/O to
 				 * one of the luns for the device at a 
 				 * time. */

@@ -1409,6 +1409,14 @@
 	sdev->lockable = sdev->removable;
 	sdev->soft_reset = (inq_result[7] & 1) && ((inq_result[3] & 7) == 2);
 
+	if (sdev->scsi_level >= SCSI_3 || (sdev->inquiry_len > 56 &&
+		inq_result[56] & 0x04))
+		sdev->ppr = 1;
+	if (inq_result[7] & 0x60)
+		sdev->wdtr = 1;
+	if (inq_result[7] & 0x10)
+		sdev->sdtr = 1;
+
 	/*
 	 * XXX maybe move the identifier and driverfs/devfs setup to a new
 	 * function, and call them after this function is called.

@@ -1509,9 +1517,9 @@
 	 * XXX maybe change scsi_release_commandblocks to not reset
 	 * queue_depth to 0.
 	 */
-	sdevscan->queue_depth = 1;
+	sdevscan->new_queue_depth = 1;
 	scsi_build_commandblocks(sdevscan);
-	if (sdevscan->has_cmdblocks == 0)
+	if (sdevscan->queue_depth == 0)
 		goto alloc_failed;
 
 	sreq = scsi_allocate_request(sdevscan);

@@ -1585,7 +1593,7 @@
 		kfree(scsi_result);
 	if (sreq != NULL)
 		scsi_release_request(sreq);
-	if (sdevscan->has_cmdblocks != 0)
+	if (sdevscan->queue_depth != 0)
 		scsi_release_commandblocks(sdevscan);
 	return SCSI_SCAN_NO_RESPONSE;
 }

@@ -1739,9 +1747,9 @@
 	if (sdevscan->scsi_level < SCSI_3)
 		return 1;
 
-	sdevscan->queue_depth = 1;
+	sdevscan->new_queue_depth = 1;
 	scsi_build_commandblocks(sdevscan);
-	if (sdevscan->has_cmdblocks == 0) {
+	if (sdevscan->queue_depth == 0) {
 		printk(ALLOC_FAILURE_MSG, __FUNCTION__);
 		/*
 		 * We are out of memory, don't try scanning any further.

@@ -2014,6 +2022,17 @@
 		 */
 		if (shost->select_queue_depths != NULL)
 			(shost->select_queue_depths) (shost, shost->host_queue);
+		if (shost->hostt->slave_attach != NULL)
+			if ((shost->hostt->slave_attach) (sdev) != 0) {
+				/*
+				 * Low level driver failed to attach this
+				 * device, we've got to kick it back out
+				 * now as a result :-(
+				 */
+				printk("scsi_scan_selected_lun: slave_attach "
+					"failed, marking device OFFLINE.\n");
+				sdev->online = FALSE;
+			}
 
 		for (sdt = scsi_devicelist; sdt; sdt = sdt->next)
 			if (sdt->init && sdt->dev_noticed)

@@ -2024,7 +2043,7 @@
 				(*sdt->attach) (sdev);
 				if (sdev->attached) {
 					scsi_build_commandblocks(sdev);
-					if (sdev->has_cmdblocks == 0)
+					if (sdev->queue_depth == 0)
 						printk(ALLOC_FAILURE_MSG,
 						       __FUNCTION__);
 				}

@@ -66,6 +66,7 @@
 EXPORT_SYMBOL(scsi_report_bus_reset);
 EXPORT_SYMBOL(scsi_block_requests);
 EXPORT_SYMBOL(scsi_unblock_requests);
+EXPORT_SYMBOL(scsi_adjust_queue_depth);
 
 EXPORT_SYMBOL(scsi_get_host_dev);
 EXPORT_SYMBOL(scsi_free_host_dev);

Please provide more details

[ linux-kernel added to the Cc: since that's where IDE stuff goes ]

OK, it repeatedly hits a BUG() during startup (after detecting the first two hard drives on the Primary IDE channel, it takes an interrupt and then during that context calls detect_ide_disk_speed or something like that, which then calls _kmem_cache_alloc during said interrupt context, resulting in the stack printout and traceback, then it continues). Second problem is that as soon as I started up /dev/md0, a raid1 array that was dirty due to all the failed boots, it almost immediately hard locks the system. That could be a raid1 hardlock, or it could be a serverworks or generic ide code lockup. Important details would be that the two mirror disks for /dev/md0 are on ide0 and ide1 so a rebuild would generate lots of primary/secondary controller parallel traffic.

I didn't provide a lot of details before because this machine doesn't have a serial console host to grab logs and I didn't write all the stuff down. Then there is the additional fubar stuff but it's mainly in the category of "Red Hat Linux 8.0 made initrd images using nash as the shell blow chunks in regards to actually working with the 2.5.40 kernel, failing simple operations such as mounting /proc, which of course causes damn near everything else it tries to do to fail since the normal /proc files aren't available".

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Ok first thing to try - does 2.4.20-pre8-ac3 also lock up. That will seperate problems with the 2.5.40 kernel from IDE ones. In paticular I notice you mention using initrd which doesnt seem to work in 2.5.40

I note that there's a lot more than dynamic queue depth adjustment in this patch (PPA inquiry, slave attach etc.).

How do HBA's that don't support this now work? You've taken any dependency on scsi_host.cmd_per_lun out of the code (thus rendering it useless) so every HBA driver now is forced to use an initial queue depth adjustment just to start tagged command queueing. Can't we at least start with cmd_per_lun as the default depth?

I'm not entirely happy with the idea that we control the queue depth by adjusting the number of the device's allocated commands. I know the patch goes to great lengths to move these kmallocs out of the critical path, but there are certain environments (multi-initiator) where the queue depth can be nastily and randomly variable. If the allocations were more lazy (wait a while before freeing a struct Scsi_Cmnd to see if the queue depth goes up again for instance) this would address some of these concerns (perhaps just moving to a slab allocator for command blocks would do it?)

I appreciate that the HBA driver is the most exact counter of the queue depth, but would it make a significant difference if the adjustments were done globally in the mid-layer? The great advantage is that we would gain dynamic queue depth adjustments without having to add specific code to every driver, at the cost of not always being entirely accurate about the depth. Is there a good argument that this really, really must be done at the LLD level given the cost in terms of LLD modifications? You could still add hooks for those HBAs that really want to do it themselves.

James

The slave attach is required for the queue depth adjustment stuff to work. The goal is to remove the select_queue_depths() function entirely and make slave_attach() -> adjust_queue_depth() the replacement. The PPR message ability flag was just a simple bit that goes along with slave attach. There's more to be done there, but that's in there as a sample.

The mid layer calls into the HBA driver's select_queue_depths() function and the driver sets the queue depth on each device from there.

This is true both before and after my patch. If the HBA doesn't implement the select_queue_depths function then it will never get tagged queueing. Besides, cmd_per_lun is *not* for tagged queueing, it's the queue depth the scsi mid layer is suppossed to keep for *non* tagged devices.

The current mid layer never has enabled a default depth, all scsi drivers must enable tagged queueing for each device before it happens. I'm just changing how that's done from a specific select_queue_depths() call in point that does one and only one thing to a generic slave_attach() call in point that sets various post-INQUIRY data drive specific items, queue depth and acceptable speed negotiation message protocols being the two examples in my patch.

It's the only safe way to do it. You can't count on being able to allocate commands because you may not have available mem, and you don't want extra laying around because that wastes mem, and it allows my next changes to go in which includes switching to linked lists for commands and that way we can have a free list for each lun and when there is a command on the free list, then you know you are under your queue depth count because if all commands are used, then none are on the free list.

Go look at the QUEUE_FULL handler in the aic7xxx_old driver. This is how most/all reasonably well written drivers handle queue depth adjustments. Trust me, they don't go around adjusting the depth all the time. Most of the time there will be one initial adjustment, then maybe one more adjustment as we lock it down to the max upper limit when one exists, the rest of the time we just handle the occasional random queue depth QUEUE_FULL messages as exactly that and only temporarily freeze the queue to let the drive get some work done.

Nope, this is a non concern because we don't call adjust_queue_depth on every queue full (or at least we shouldn't, there isn't any documentation yet to tell people that, but that was the purpose of making the aic7xxx_old driver implement the new method since it serves as an example and it does exactly as I'm talking about).

The mid layer simply does not have access to the info needed to do what you are talking about. Let me give you an example from the aic7xxx driver. On this card we have a sequencer that handles the scsi protocol for us. When we get a command from the mid layer it goes either to the card's QINFIFO (queue of commands that the sequencer hasn't started yet) or to the device's waiting queue (queue of commands that we can't send to the sequencer yet because of some reason). Once it is on the card and the sequencer starts the process of selecting the target device, the command is moved to the waiting_q on the card (queue of commands already started but for which the target device has not yet responded). Once the device is selected and we get to send the command, then we usually go directly into status phase and get our QUEUE_FULL. At this point the sequencer pauses all operations on the scsi bus and interrupts the kernel to handle the QUEUE_FULL. Prior to this command being selected, there may have been any number of commands place in the cards QOUTFIFO (our queue of already completed SCSI commands which the kernel interrupt driver hasn't processed yet). So when the kernel driver gets the QUEUE_FULL interrupt, it must first plug this device up. It then must remove any commands it finds in the QINFIFO and in the waiting_q on the card and requeue them back into the device's waiting queue. Each command it moves like this decrements the device's active command count since when we place a command on the QINFIFO that's when we increment the active count. Then we have to pull any possible completed commands out of the QOUTFIFO and place them on the done queue (commands we have queued to go back to the mid layer), also decrementing the active count each time. Finally, we decrement the active count for the device one more time to accomodate our QUEUE_FULL command. Only after we have done *all* of this crap do we have an accurate queue depth count for the command that returned a QUEUE_FULL. As you can see, lots of this is very hardware specific. Controller cards that don't have a sequencer like control engine won't be starting commands off in a lazy fasion like this and so don't need all this stuff. That's why the only place where we can get an accurate count of the queue depth is in the low level driver. So, that code *must* stay there since getting an accurate count is how we avoid the random flip flops of queue depth that you were worried about earlier.

Now, what can be moved to the mid layer, and is on my list to do, is a generic interface for coming up with the proper action after a QUEUE_FULL. Currently, each driver not only determines the real depth, but then also does it's own magic to tell if it's a random event or a hard limit. It would be easy to add something like scsi_notify_queue_full(sdev, count); where scsi_notify_queue_full() would keep track of the last queue full depth, etc. Let the low level driver come up with the accurate count, then they can use this function to determine what to do about it. On any change to queue depth, the function can return the amount of commands to add/subtract, then the low level driver can adjust it's own internal structure counts and also call scsi_adjust_queue_depth() to have the mid layer do likewise. BTW, I'll change the adjust_queue_depth code to make it immediately adjust the depth down when possible and do lazy increases so that hitting a hard limit will free up resources immediately, but that will go with making the commands linked list based so that it can simply do a while(sdev->queue_depth > sdev->new_queue_depth && list_not_empty(sdev->free_list)) { kfree(get_list_head(sdev->free_list)); sdev->queue_depth--; }

You're never going to get around this and have a reasonably reliable method :-(

Well, all the device drivers that implement queue depth adjustments at all already do it themselves, so my proposed method is better than what we currently have since it at least allows them to move the decide disposition stuff into a library call or they can do it themselves.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

But that's not quite how it works today.

If a HBA driver doesn't implement select_queue_depths, it can still set the template cmd_per_lun and have the mid layer build it this many commands per device (that's the line in scsi_build_commands which sets queue_depth to cmd_per_lun if its still unset), and hence you have an effective default tagged queue depth of cmd_per_lun.

The patch will break this, and thus disable TCQ for all drivers that rely on it.

The comment in the code above this parameter says: "...Set this to the maximum number of command blocks to be provided for each device. Set this to 1 for one command block per lun, 2 for two, etc." It's not unreasonable to assume this means the maximum number of outstanding tags.

Even if we assume it means what you say, how does the non TCQ HBA that sets this to 2 so every device always has one outstanding command and one ready to roll work? Now it only gets one command block because we never consult cmd_per_lun.

James

OK, I spent a nice evening doing this (do I get brownie points?). I see your algorithm is roughly lower the depth after 14 queue fulls and assume that luns of the same pun need to be treated equivalently.

I failed entirely to find how the queue depth is increased (no brownie points here). How is that done?

[...]

Yes, many HBAs have internal "issue" queues where commands wait before being placed on the bus. I was assuming, however, that when a HBA driver got QUEUE FULL, it would traverse the issue queue and respond QUEUE FULL also to all pending commands for that device. The mid-layer should thus see a succession of QUEUE FULLs for the device (we even have a nice signature for this because the QUEUE FULL occurs while device_blocked is set). However, as long as it can correctly recognise this, it knows that when the last QUEUE FULL is through it has the true device queue depth, doesn't it?

I'll go for this. That would address my main concern which is a proliferation of individual queue full handling algorithms in the LLDs. (and it's better than teaching the mid-layer about QUEUE FULL sequences).

Fair enough.

James

After fixing up the 53c700 to use the new adjust queue depths stuff, I can confirm it's working fine for me here (well, actually I fixed the generic scsi tcq code to use it, but only the 53c700 driver uses this).

James

===== scsi.h 1.22 vs edited =====

@@ -882,7 +882,7 @@
 
         if(SDpnt->tagged_supported && !blk_queue_tagged(q)) {
                 blk_queue_init_tags(q, depth);
-                SDpnt->tagged_queue = 1;
+		scsi_adjust_queue_depth(SDpnt, 1, depth);
         }
 }
 

@@ -892,7 +892,7 @@
  **/
 static inline void scsi_deactivate_tcq(Scsi_Device *SDpnt) {
         blk_queue_free_tags(&SDpnt->request_queue);
-        SDpnt->tagged_queue = 0;
+	scsi_adjust_queue_depth(SDpnt, 0, 2);
 }
 #define MSG_SIMPLE_TAG	0x20
 #define MSG_HEAD_TAG	0x21

(Note: the aic7xxx_old driver has flawed pun/lun tagged queue differentiation. I know this. It's not trivial to fix and I'm not going to fix it until I send in the patch that implements decision making at the library level. I have no plan to ever fix it in the actual aic7xxx_old driver, only in the library) Yes, you get brownie points ;-)

It isn't. When we set the queue depth during init, we set it to the maximum allowed. We only reduce the depth after getting 14 consecutive QUEUE_FULLs at the same depth. By that point in time we can safely assume that the 14 QUEUE_FULLs represented a hard limit, not a transient limit, and therefore there is no need to ever raise the limit again. Keep in mind that in the driver, every time we get a QUEUE_FULL, we freeze the queue until either A) a command completes or B) 10ms have passed. That means we won't get 14 QUEUE_FULLs without the drive having a chance to change the current situation and remedy any transient conditions.

Nope. It *might* be possible to make this work, but there is no guarantee as it stands. Consider that while processing the QUEUE_FULLs I may have other successfully completed commands on my done queue waiting to be sent to the mid layer. Furthermore, they get queued for a tasklet to handle. What happens if a driver sends the QUEUE_FULLs first and then the completed commands next? We get a wrong count. In general, you are suggesting that we impose a strict syncronization between the mid layer and the low level driver, including ordering restrictions and more, when instead we could just call one function with our final count once we know it. We still have to do all the work in the low level driver of finding the commands anyway, so my way of doing things only adds a few lines of code to each driver beyond what they must have regardless and it avoids the syncronization headache this other method brings up.

Good, then we agree 100% then ;-)

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Excellent ;-) However, a couple questions.

1) When/How is the lldd notified that tagged queueing has been enabled and the depth to which it is enabled?

2) How does the lldd tell the midlayer (and hence the block layer) to reduce the queue depth?

Content-Description: tmp.diff

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

A trick question. Obviously it should be in slave_attach. However, the 53c700 is used with a lot of quirky drives, so the way I do it is to shadow the commands. The first command that comes down with tagged_supported set, I try a tagged command. If that command succeeds, I then activate the TCQ and thus set tagged_enabled (the driver has several bits per target to support this state model). If the first tag fails, I clear tagged_supported. This catches the annoying drives that set the CmdQue bit in the inquiry page and then reject all tagged commands. The depth for the 53c700 is just the default cmd_per_lun.

You can see this in the source code (tag_negotiated bitmap and NCR_700_DEV_BEGIN_TAG_QUEUEING flag).

It can't, currently. This could be done, but only by disabling and reenabling the block layer TCQ support. The block layer currently uses a static array for this.

However, the resources the block layer uses are tiny (one entry in this array per tag, plus a bitlist) so its probably just not worth freeing and reacquiring these. To integrate with the blk layer, I'd propose just initialising the blk layer with the maximum possible number of tags (i.e. introduce an upper limit somewhere) and then just do the dynamic stuff in the mid-layer.

James

The Qlogic host adapters do not provide a way to find the true queue depth to the device as described previously by Doug. The midlayer depth is probably about as accurate as you could get, with the exception being if the Qlogic driver runs out of IOCBs. I don't believe that they return Queue Full for unsubmitted commands, but I do not know for sure. I certainly would not expect every host adapter to do that.

That was one of the holes in the Ordered Tag scheme. You issue a bunch of unordered commands, then an ordered one, followed by a bunch of unordered.

If the ordered tag gets the Queue Full, and you are depending on the Ordered semantics, you're in Big Trouble.

jeremy

Is anyone still working on this code or have the changes other people wanted to make all been integrated?

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

I have some changes to throw most of it away and re-implement it in user space using hotplug instead. I think this type of thing is a little radical for the eve of 2.6, so feel free to alter scsi_scan.c in any way you want.

James

Not that I know of.

I never re-sent patches to change the GFP_ATOMIC to GFP_KERNEL. With the sleep debugging, and per other discussions, no in_interrupt check should be needed. I can create, test, and send them to you or the list, but the change is simple.

More device attributes should be added, so we can move away from /proc/scsi - like vendor, product, online, and maybe more.

The device model "name" should be replaced for now with our own "uid" field, and name used as a generic description, probably the same thing that is put into the type field right now, or something for disk like "scsi direct-access" and for tape "scsi sequential-access".

-- Patrick Mansfield

(Not a bad plan, I rather like that idea myself)

OK, then I'm working on it as soon as my 2.5 test box works right again (ServerWorks IDE seems broke on my box and I don't have the time to fix it when I've got this other stuff waiting).

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Sorry, I did not include SCSI-2 LUN setting changes for scsi_scan.c in my recent consolidate SCSI-2 command lun setting patch, I re-rolled the patch with scsi_scan.c changes, and I'm asking James to push it.

Also, I was excluding my multi-path changes - I'm assuming I'll have to merge with whatever changes you make.

-- Patrick Mansfield

The last change to ips was you on 16 October, so it's probably still AWOL.

I'm keeping the trivial miscellaneous changes in linux-scsi.bkbits.net/scsi-mis c-2.5 you can look at the change logs (even if you don't run bitkeeper) to see what's in there.

James

This is really exposing our whole detect init attach finish slave_attach mess which Christoph was trying to clean up.

What happens now (after Christoph's removal of finish) is:

1) call detect to see if anyone will attach. If yes, call slave attach (we adjust the queue depth here if wanted)
2) call init (if not inited---done once per upper level driver) --- init is supposed to be for first init of the SCSI upper level device driver to allocate inital resources, so we already have detect called befor init.
3) call attach but release the commandblocks if sdev->attached comes back zero.

-> attached, by the way is a semaphore because we can have more than one upper level attachment. This exposes nicely in the rather icky algorithm of scsi_get_request_dev() which is used to find which upper level driver should process the request in the request function.

It looks like what we want to have happen is:

1) init first
2) detect is currently superfluous (there will probably be a future time when we need a separate detect for lazy attachment---devices don't attach and consume resources until the operator so instructs)
4) call build commandblocks to get a single working command
5) attach
6) if anything attached do the slave attach or cmd_per_lun adjustment. If not, release the command blocks.

The net effect should be that if a device has no upper level driver, it can appear in /proc/scsi/scsi, but it's consuming no other system resources.

James

Well, I'm working on removing ->init. With the patch posted yersterday only sg.c and osst.c are left implementing it.

Completly agreed with that.

We should also call blk_cleanup_queue along with scsi_release_commandblocks, and then call scsi_initialize_queue when we call scsi_build_commandblocks in scsi_register_device (or wherever we have the corresponding scsi_build_commandblocks call).

The request queue memory usage is probably a lot higher than what is used for one Scsi_Cmnd.

-- Patrick Mansfield

Ermm, osst (which I maintain) works the same as st in this respect. I didn't notice any changes to st.c in that patch. What am I missing?

Thanks, Willem Riede.

st in 2.5 doesn't have ->init and ->finish anymore. Time for a resync? :)

Obviously. I'll do that. I have to admit I've kind of ignored 2.5.x to let it stabilize before spending effort on keeping osst up to date, but it looks like it's time to make up for lost time...

Thanks, Willem Riede.

I agree.

I agree. (That's what patrick just put in the patch for me).

Quite like the idea. It should be possible to make it complement the request_prep_fn I'm doing for SCSI (this will force the upper level drivers to do early prepares on commands, so we can have the whole thing just waiting to shoot off as the completed command returns).

James

I'm just getting ready to add Patrick's patch to scsi-for-linus-2.5 and bring it up to date with the BK current. Should be about another hour (depending on how long it takes me to compile and test)

James

There's a bunch of useless mod usecount handling in scsi:

o MOD_INC_USE_COUNT/MOD_DEC_USE_COUNT in exported functions that are only used outside this module - a module that has exported symbols use by other modules can't be unloaded at all
o GET_USE_COUNT checks in scsi_unregister_device/scsi_unregister_host - the scsi layer shouldn't care about the mod use count of the modules implementing drivers, and as they are usually called from the mod unload handler the check is totally pointless (if it ever triggers the scsi midlayer could try to scramble at freed memory). Also GET_USE_COUNT is disfunct with the new module loader
o lacking ->owner in megaraid's chardev leading to junk
o double-increment of the modules use count in upper layer drivers (chardev/bdev open already gets a reference on ->owner for us)

I also had a patch to get the increment on the host driver module in upper layer ->open right, but as Rusty's new loader changed the interface for that I'll submit that one once the new loader appears in the scsi tree.

===== drivers/scsi/NCR53C9x.c 1.12 vs edited =====

@@ -732,9 +732,6 @@
 	/* Reset the thing before we try anything... */
 	esp_bootup_reset(esp, eregs);
 
-#ifdef MODULE
-	MOD_INC_USE_COUNT;
-#endif
 	esps_in_use++;
 }
 

@@ -3638,7 +3635,6 @@
 void cleanup_module(void) {}
 void esp_release(void)
 {
-	MOD_DEC_USE_COUNT;
 	esps_in_use--;
 	esps_running = esps_in_use;
 }

===== drivers/scsi/hosts.c 1.27 vs edited =====

@@ -220,12 +220,6 @@
 	struct scsi_cmnd *scmd;
 
 	/*
-	 * Current policy is all shosts go away on unregister.
-	 */
-	if (shost->hostt->module && GET_USE_COUNT(shost->hostt->module))
-		return 1;
-
-	/*
 	 * FIXME Do ref counting.  We force all of the devices offline to
 	 * help prevent race conditions where other hosts/processors could
 	 * try and get in and queue a command.

@@ -520,8 +514,6 @@
 
 	cur_cnt = scsi_hosts_registered;
 
-	MOD_INC_USE_COUNT;
-
 	/*
 	 * The detect routine must carefully spinunlock/spinlock if it
 	 * enables interrupts, since all interrupt handlers do spinlock as

@@ -604,8 +596,6 @@
 	if (pcount != scsi_hosts_registered)
 		printk(KERN_INFO "scsi : %d host%s left.\n", scsi_hosts_registered,
 		       (scsi_hosts_registered == 1) ? "" : "s");
-
-	MOD_DEC_USE_COUNT;
 
 	unlock_kernel();
 	return 0;

===== drivers/scsi/megaraid.c 1.27 vs edited =====

@@ -760,9 +760,8 @@
 /* For controller re-ordering */ 
 
 static struct file_operations megadev_fops = {
-	ioctl:megadev_ioctl_entry,
-	open:megadev_open,
-	release:megadev_close,
+	.owner		= THIS_MODULE,
+	.ioctl		= megadev_ioctl_entry,
 };
 
 /*

@@ -4333,15 +4332,6 @@
 	}
 }
 
-/*
- * Routines for the character/ioctl interface to the driver
- */
-static int megadev_open (struct inode *inode, struct file *filep)
-{
-	MOD_INC_USE_COUNT;
-	return 0;		/* success */
-}
-
 static int megadev_ioctl_entry (struct inode *inode, struct file *filep,
 		     unsigned int cmd, unsigned long arg)
 {

@@ -4851,16 +4841,6 @@
 
 	return scb;
 }
-
-static int
-megadev_close (struct inode *inode, struct file *filep)
-{
-#ifdef MODULE
-	MOD_DEC_USE_COUNT;
-#endif
-	return 0;
-}
-
 
 static int
 mega_support_ext_cdb(mega_host_config *this_hba)

===== drivers/scsi/osst.c 1.27 vs edited =====

@@ -4176,8 +4176,6 @@
 
 	if (STp->device->host->hostt->module)
 		 __MOD_INC_USE_COUNT(STp->device->host->hostt->module);
-	if (osst_template.module)
-		 __MOD_INC_USE_COUNT(osst_template.module);
 	STp->device->access_count++;
 
 	if (mode != STp->current_mode) {

@@ -4520,8 +4518,6 @@
 
 	if (STp->device->host->hostt->module)
 	    __MOD_DEC_USE_COUNT(STp->device->host->hostt->module);
-	if (osst_template.module)
-	    __MOD_DEC_USE_COUNT(osst_template.module);
 
 	return retval;
 }

@@ -4651,8 +4647,6 @@
 
 	if (STp->device->host->hostt->module)
 		__MOD_DEC_USE_COUNT(STp->device->host->hostt->module);
-	if(osst_template.module)
-		__MOD_DEC_USE_COUNT(osst_template.module);
 
 	return result;
 }

===== drivers/scsi/scsi.c 1.61 vs edited =====

@@ -2404,8 +2404,6 @@
 		}
 	}
 
-	MOD_INC_USE_COUNT;
-
 	if (out_of_space) {
 		scsi_unregister_device(tpnt);	/* easiest way to clean up?? */
 		return 1;

@@ -2422,12 +2420,6 @@
 	struct Scsi_Device_Template *prev_spnt;
 	
 	lock_kernel();
-	/*
-	 * If we are busy, this is not going to fly.
-	 */
-	if (GET_USE_COUNT(tpnt->module) != 0)
-		goto error_out;
-
 	driver_unregister(&tpnt->scsi_driverfs_driver);
 
 	/*

@@ -2458,16 +2450,12 @@
 		prev_spnt->next = spnt->next;
 	up_write(&scsi_devicelist_mutex);
 
-	MOD_DEC_USE_COUNT;
 	unlock_kernel();
 	/*
 	 * Final cleanup for the driver is done in the driver sources in the
 	 * cleanup function.
 	 */
 	return 0;
-error_out:
-	unlock_kernel();
-	return -1;
 }
 
 #ifdef CONFIG_PROC_FS

===== drivers/scsi/sd.c 1.91 vs edited =====

@@ -453,8 +453,6 @@
 	 */
 	if (sdp->host->hostt->module)
 		__MOD_INC_USE_COUNT(sdp->host->hostt->module);
-	if (sd_template.module)
-		__MOD_INC_USE_COUNT(sd_template.module);
 	sdp->access_count++;
 
 	if (sdp->removable) {

@@ -498,8 +496,6 @@
 	sdp->access_count--;
 	if (sdp->host->hostt->module)
 		__MOD_DEC_USE_COUNT(sdp->host->hostt->module);
-	if (sd_template.module)
-		__MOD_DEC_USE_COUNT(sd_template.module);
 	return retval;	
 }
 

@@ -536,8 +532,6 @@
 	}
 	if (sdp->host->hostt->module)
 		__MOD_DEC_USE_COUNT(sdp->host->hostt->module);
-	if (sd_template.module)
-		__MOD_DEC_USE_COUNT(sd_template.module);
 
 	return 0;
 }

===== drivers/scsi/sg.c 1.39 vs edited =====

@@ -1304,8 +1304,6 @@
 				if (sdp->device->host->hostt->module)
 					__MOD_DEC_USE_COUNT(sdp->device->host->hostt->module);
 			}
-			if (sg_template.module)
-				__MOD_DEC_USE_COUNT(sg_template.module);
 			sfp = NULL;
 		}
 	} else if (srp && srp->orphan) {

@@ -1529,8 +1527,6 @@
 				}
 				if (sfp->closed) {
 					sdp->device->access_count--;
-					if (sg_template.module)
-						__MOD_DEC_USE_COUNT(sg_template.module);
 					if (sdp->device->host->hostt->module)
 						__MOD_DEC_USE_COUNT(
 							sdp->device->host->

@@ -2522,8 +2518,6 @@
 		sfp->closed = 1;	/* flag dirty state on this fd */
 		sdp->device->access_count++;
 		/* MOD_INC's to inhibit unloading sg and associated adapter driver */
-		if (sg_template.module)
-			__MOD_INC_USE_COUNT(sg_template.module);
 		if (sdp->device->host->hostt->module)
 			__MOD_INC_USE_COUNT(sdp->device->host->hostt->module);
 		SCSI_LOG_TIMEOUT(1, printk("sg_remove_sfp: worrisome, %d writes pending\n",

===== drivers/scsi/sr.c 1.65 vs edited =====

@@ -130,8 +130,6 @@
 	cd->device->access_count--;
 	if (cd->device->host->hostt->module)
 		__MOD_DEC_USE_COUNT(cd->device->host->hostt->module);
-	if (sr_template.module)
-		__MOD_DEC_USE_COUNT(sr_template.module);
 }
 
 static struct cdrom_device_ops sr_dops = {

@@ -473,8 +471,6 @@
 	cd->device->access_count++;
 	if (cd->device->host->hostt->module)
 		__MOD_INC_USE_COUNT(cd->device->host->hostt->module);
-	if (sr_template.module)
-		__MOD_INC_USE_COUNT(sr_template.module);
 
 	/* If this device did not have media in the drive at boot time, then
 	 * we would have been unable to get the sector size.  Check to see if

===== drivers/scsi/wd33c93.c 1.5 vs edited =====

@@ -1853,7 +1853,6 @@
    printk("\n");
    printk("           Version %s - %s, Compiled %s at %s\n",
                WD33C93_VERSION,WD33C93_DATE,__DATE__,__TIME__);
-   MOD_INC_USE_COUNT;
 }
 
 

@@ -2031,7 +2030,6 @@
 #endif
 void wd33c93_release(void)
 {
-   MOD_DEC_USE_COUNT;
 }
 
 MODULE_LICENSE("GPL");

[ snipped patch ]

This patch conflicts with one I had internally that I was still working on. I had a bit more stuff in mine than you did in yours so I folded your changes into my stuff. When I push my tree out it will be in there (otherwise it would have required hand merging by someone else anyway). I just gotta wait till I go back up to the office and restart my test box in order to push the changes out :-/ (shutdown -nr segfaulted on me)

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

While playing with scsi_debug today I saw the opposite effect:
with a file system mounted on a scsi_debug fake disk **, lsmod showed a zero "in use" count. I was able to "rmmod scsi_debug" and crash the system when I tried to access that mounted fs.

** A further problem I am seeing is building partitions with fdisk (on scsi_debug [haven't tried a real disk]) that don't get reflected in /proc/partitions . I can add a partition to /dev/sda but 'mke2fs /dev/sda1' fails with "no such device". BTW I can forget partitions and successfully work with the whole disk (i.e. "mke2fs -F /dev/sda ; mount /dev/sda /mnt/x"). Lk 2.5.47 on RH8.0 system with fdisk from util-linux-2.11r-10 .

Doug Gilbert

This change didn't change the use-count handling for HBA driver at all, so this must be something different. Do you use a scsi_debug using scsi_add_host/scsi_remove_host? Try adding

	.module		= THIS_MODULE,

to sdebug_driver_template, I forgot that change when sendig you the initial draft version of the change.

Yes.

That fixed it. [Still have "missing" partition(s) problem.]

Thanks. New version of scsi_debug coming soon.

Doug Gilbert

OK, I'll back this one out of the scsi-misc-2.5 tree I have internally before putting it out on the bkbits site.

James

Cool. I'm up here working on it at the moment. I'm also trying to track down the problem Doug Gilbert reported and that I was seeing last night as well, aka on /dev/sdc I can access /dev/sdc but not /dev/sdc1 which is a valid partition. Once I've tracked that down I'll push my tree up.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

So, this is the change I made to update it so that it won't puke on large devices. It seems to be working OK for me, but I didn't check the gcc assembly output to see if it's horrible. Unless someone tells me I'm smoking crack for making this change, I'll submit it pretty soon (I don't consider myself a partition expert, I'm just trying to make sure that when geometry does get defined on my driver that it at least is something the Adaptec BIOS will accept, and the Adaptec BIOS *only* accepts the two listed head/sector combos).

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.878 -> 1.878.1.1
# drivers/scsi/aic7xxx_old.c 1.32 -> 1.33
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/11/17 [email protected] 1.878.1.1
# aic7xxx_old: fix up the biosparam function to do 64bit math safely # --------------------------------------------
#

diff -Nru a/drivers/scsi/aic7xxx_old.c b/drivers/scsi/aic7xxx_old.c
@@ -10974,7 +10974,8 @@
 aic7xxx_biosparam(struct scsi_device *sdev, struct block_device *bdev,
 		sector_t capacity, int geom[])
 {
-  int heads, sectors, cylinders, ret;
+  sector_t heads, sectors, cylinders;
+  int ret;
   struct aic7xxx_host *p;
   unsigned char *buf;
 

@@ -10991,18 +10992,26 @@
   
   heads = 64;
   sectors = 32;
-  cylinders = (unsigned long)capacity / (heads * sectors);
+  cylinders = capacity >> 11;
 
   if ((p->flags & AHC_EXTEND_TRANS_A) && (cylinders > 1024))
   {
     heads = 255;
     sectors = 63;
-    cylinders = (unsigned long)capacity / (heads * sectors);
+    /* pull this crap because 64bit math in the kernel is a no-no as far
+     * as division is concerned, but 64bit multiplication can be done */
+    /* This shift approximates capacity / (heads * sectors) */
+    cylinders = capacity >> 14;
+    /* Now we brute force upping cylinders until we go over by 1 */
+    while( capacity >= (cylinders * sectors * heads))
+      cylinders++;
+    /* Then back it back down by one */
+    cylinders--;
   }
 
-  geom[0] = heads;
-  geom[1] = sectors;
-  geom[2] = cylinders;
+  geom[0] = (int)heads;
+  geom[1] = (int)sectors;
+  geom[2] = (int)cylinders;
 
   return (0);
 }

Well, SCSI doesn't implement the BIG_GETGEO ioctl, so this isn't really much use. You can use sector_div to do the divisions.

However, Andries Brower assures me that the geometry stuff is unused (at least from the cylinders point of view), so all of this is unnecessary anyway. There's a large thread about it on linux-scsi (don't have the reference). The upshot was the code you can see in scsicam.c:scsicam_bios_param().

So the short answer is that this should be unnecessary.

James

Yes. But note: the number of cylinders (i) is totally immaterial, (ii) is returned in a short, the only place where it is used. Thus, computing values larger than 65535 is undesired, they will be truncated later.

That is why you can replace this strange

	if (capacity > 65535 * sectors * heads)
		cylinders = 65535;
	else
		cylinders = ((unsigned int) capacity) / (sectors * heads);

What about using sector_div ?

Didn't know it existed. Pointer?

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Hmmm..should that be changed though?

I beg to differ. At least on the Adaptec stuff, it gets pissy if the partition tables don't look sane (at least older BIOSes do, and those cards are still in use, so that means I try and make them work). Besides, when I make changes in this code, the effects are instantly visible in fdisk (well, after a module reload), so it get's used somewhere.

No, it's needed. If it isn't there, then when a disk is currently empty of any partitions, the generic geometry stuff picks out whatever it can find that fits the total capacity best without regard to the BIOS issues on the older Adaptec cards, resulting in funky crap that doesn't work.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

That is a bit of information I was looking for. I figured it would get truncated somewhere (or worse, wrap), and I wanted to know where that might be.

Thanks, I'll do that.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

I hope not. If we implement BIG_GETGEO then I've made all the wrong decisions for biosparam: since it can pass back ints, the implementations should all be using sector_div and we should do the truncation in the GETGEO ioctl since we can use the same biosparam call for both ioctls.

The argument is that no user space tools rely on the cyls value, only the sectors and heads. They all use the GETGEO ioctl, but also get the blocksize of the device and divide this out by the sectors*heads to get the true cyls value.

I know this is the wrong short term decision: you should never implement an interface wrongly because it causes exactly this type of confusion. I'm just betting on the C/H/S mess being sorted out soon....I hope.

James

That may very well be true and I'm all for that, but I'm still anal about being right anyway, so I implemented what Andries suggested.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

Erm, well, smaller patches and earlier incorporation? :-)

I was just about to. I have to resync scsi-misc-2.5 with BK current first. If you can give me an hour, I'll have it done.

James

It's a race Charlie Brown! :-P

Thanks!

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

The scsi-misc-2.5 resync should be done now.

James

Hi -

hardirq.h and init.h were removed from scsi.h. So I had to make the following changes to compile. I also removed two extraneous includes.

@@ -12,6 +12,7 @@
 #include <linux/bio.h>
 #include <linux/kernel.h>
 #include <linux/blk.h>
+#include <asm/hardirq.h>
 #include <linux/smp_lock.h>
 #include <linux/completion.h>
 

===== drivers/scsi/scsi_proc.c 1.10 vs edited =====

@@ -16,14 +16,13 @@
  * Michael A. Griffith <[email protected]>
  */
 
-#include <linux/config.h>
 #include <linux/module.h>
+#include <linux/init.h>
 #include <linux/string.h>
 #include <linux/mm.h>
 #include <linux/slab.h>
 #include <linux/proc_fs.h>
 #include <linux/errno.h>
-#include <linux/stat.h>
 #include <linux/blk.h>
 #include <asm/uaccess.h>
 

-- Patrick Mansfield

There's only a single outstanding change against the Linus tree currently. It's at http://linux-scsi.bkbits.net/scsi-misc-2.5

James

On Oct 27, 11:13am, Doug Ledford wrote:
} Subject: Re: RFC - device names and mdadm with some reference to udev.

Good evening to everyone, hope the week has gone well.

Whatever we do please do not make use of mdadm or startup of arrays dependent on udev. I do SAN's for a living and have had far too many phone calls and have spent too much time trying to get boxes messed up by udev back on the fabric to want to add any more complication to the mix.

The notion of udev certainly has its place but not on a server which only cares about four device nodes for its entire operational life.

Neil your mdadm is a great tool and your contributions via the MD stuff are beyond peer, keep up the good work. But this stuff has to get simpler rather than more complex.

Best wishes for a pleasant weekend to everyone.

}-- End of excerpt from Doug Ledford

As always,

Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.

PH: 701-281-1686
FAX: 701-281-3949 EMAIL: [email protected]
------------------------------------------------------------------------------
"C++ is designed to allow you to express ideas, but if you don't have
any ideas or don't have any clue about how to express them, C++ doesn't offer much help."
-- Bjarne Stroustrup
Technology Review

I had intended to continue to support the no-udev installations, but thank you the encouragement that it really is needed and will be used.

Just a clarification: are you envisaging an installation without udev at all, or one with udev installed and active, but you don't wont mdadm to depend on it? That latter option may be more awkward (I currently support an environment variable which says "just create the devices, even if udev appears to be installed").

Thanks :-)

NeilBrown

+1

i have no objections in letting udev create my device files. what i dislike is when udev sees a device appearing, and decides it knows better than me, so it starts using it right away, no matter if it is an usb key or a multi-tb shared storage.
I could never have tought of something so stupid as the incremental assembly of md arrays,
0 advantages gained for lot of trouble.

L.

Luca Berra -- [email protected]
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \