scsi command slab allocation under memory pressure

James had a similiar comment about this (free_list storing multiple commands).

The linux-scsi.bkbits.net scsi-kmem_alloc-2.5 and scsi-combined-2.5 tree include the scsi command slab allocation (Luben's patch).

How does the use of a single slab for all hosts and all devices allow for IO while under memory pressure?

There is one extra scsi command pre-allocated per host, but don't we require at least one (and ideally maybe more) per device? The pre-slab (current mainline kernel) command allocation always had at least one command per device available, and usually more (because we allocated more commands during the scan and upper level init).

That is - if we have swap on a separate disk and our command pool is small enough, IO to another disk could use the single per-host command under memory pressure, and we can fail to get a scsi command in order to write to the swap disk.

scsi_put_command() re-fills the host->free_list if it is empty, but under high (or higher) IO loads, the disk/device that generated the scsi_put_command will immediately issue a scsi_get_command for the same device.

If all command allocations are failing for a particular device (i.e. swap), we will wait a bit (device_blocked and device_busy == 0) and try again, we will not retry based on a scsi_put_command(). Even if we did retry based on a scsi_put_command, we will can race with the scsi_put_command caller.

-- Patrick Mansfield

Is this a question, narrative, comment or flame? I'll try to answer this anyway.

James had this comment, just because I put the mechanism there to allow for more than one command to be in the store of backup command structs. See my reply to his email in the archives.

The reason for populating free_list with just one command on host init, is quite obvious, but to elaborate, the choices are 1 and N, where N is a natural number greater than 1.

The problem with N is that I do *not* have a heuristic which will tell me what a suitable value for N is. How about 5, hmm, what about 10, or maybe 1e10?

Furthermore, N may be a constant or it may be a function of how much memory we currently have, how many commands have been queued into the host, etc, or it may just be N = can_queue - num_queued_commands + 1, which is dynamic, which is pointless (Homework: show why).

We want to waste as little memory as possible (thus 1 per host), since SCSI Core is not the only subsystem running on the machine. Hint:
see the flags the slabs are allocated with. Using the Central Limit Theorem, I hope that by the time we get low on memory pressure, the scsi command cache pool size has settled*. Unless we started with very little memory, which would be quite unusual in this day and age.

* Lots of assumptions here, but all valid for a *server* machine.

Let's get some experience with this thing running and actually have a *natural* failing example, and we can twiddle with the initial value of N, and/or can develop a f(N) which would be computed occasionally and free_list varied upon scsi_put_command().

Luben

P.S. In my own mini-scsi-core I haven't had any problems with this issue.

Yes, except for the flame part :)

We do have bounds on the number of commands.

Assuming N is the minimum number of commands we want available:

Currently (plain 2.5.59) we have N being at the number of scsi_device's on the system, and we always have at least one command available for each scsi_device.

It is not clear if N should be the number of scsi commands that might be outstanding on the system (what it used to be prior to Doug L's queue depth changes, and what we have in 2.4).

So N should be at least the number of scsi_devices on the system, and at most the sum of all the commands that can potentially be used by all scsi_devices on the system.

If we must allocate enough space to ensure proper swapout behavour, it is not a waste!

IMO we need something similiar to the request_queue_t free list, where we allocate a bunch of items up front out of a slab.

Are you saying we will always be able to get a command under low memory situtations?

If not, the code should be changed, otherwise please explain how.

We should not wait for a failure to occur to answer the question.

-- Patrick Mansfield

Yes, and it is can_queue and it is per host -- we all know this. I was talking about N, which is a bit different matter.

This has two problems: 1. ``currently'' and 2. number of scsi_devices on the system.

1. We should improve, ``currently'' may not be the best policy. (Thus the slab allocator.)

2. If I had to go this route, I'd say: the number of scsi_devices per host; but this number is dynamic as devices come and go. (So we get a hint at a dynamic heuristic.)

This has two problems: 1. ``It is not clear'' and 2. ``outstanding on the system''.

1. I think I did mention this in my previous reply. (Thus N = 1.)

2. This is a pickle and might I mention that Doug's queue depth changes were *for a reason*, so there's no point in saying ``the way it was before''. Furthermore ``the number of outstanding commands'' is ambiguous and any which way it is it, is NOT enough. I.e. outstaning in LLDD, oustanding free, etc. -- it just doesn't compute.

How did you deduce this if one of your premises starts with ``It is not clear''?

This has 2 problems: 1. ``number of scsi_devices on the system'' and
2. ``sum off all commands which can potentially be used by all scsi_devices on the system''.

1. This number is dynamic. N is *per host* and not per SCSI Core, i.e. it would've been more proper to say, ``number of scsi devices per host'', which would also not be sufficient, since this number is also dynamic.

Furthermore, if N > 1, and the system is under memory pressure such that the cache allocator fails, it is true that a single device can starve the others of scsi commands, for any 1 < N <= can_queue ! So this doesn't quite do it as well, and a per device back-up would be needed.

Which would *not* be that helpful since we can get in to the same argument.... how many is good enough.

2. In which case we *do not* need the cache allocator at all. I.e. you're saying that N = can_queue - number_cmnds_queued + 1, and I did comment on this in my previous mail and will say it
again: in which case we do NOT need the cache allocator at all. (homework: show why)

And the whole reason of this is to give memory allocation to another subsystem. Hint again: see the flags with which the caches are created.

Are we creating a problem with generalims like this one? How much is ``enough''? Or how long is the long hair?

I think I mentioned this conundrum in my previous email.

Yes, this a static heuristic. See below for comments on this policy.

The cache allocator page values should've settled after a few hours of intensive SCSI IO. (Cf. Central Limit Theorem) And those values always *round up* by the lookaside cache, which means that the cache allocator would always have more than can_queue if in that time can_queue was ever reached.

It would be *quite a rare* circumstance that the cache allocator would fail (again hint: see the flags which with they are created), and even if it did fail, chances are that other subsystems would free memory, or a scsi command would finish, etc. in which case SCSI Core could go on.

It could fail if, e.g., an exorbitant amount of scsi devices were plugged on the fabric *while* the system was experiencing memory pressure. Thus the above mentioning average value would not be correct anymore, as the number of devices has changed. *BUT* by having N=1, we wouldn't load the system as much as if we had a different initial N, say N=10.

So this is tricky, it could be the case that in two exact same circumstances when N=1 the system could survive, but for N=10, we crash it.

So the best policy is a dynamic heuristic.

But I think that *YOU* should be making this argument, since it looks like you're not happy with the slab allocator being used for scsi commands.

Looks like you have an agenda. If you have nothing else to do, please go ahead and rip the code apart and do your own thoughtful and smart changes. Go ahead, overengineer as you wish.

Another generalism.

If you have a heuristic for N, either *static*, i.e.

	N = can_queue/10 + 1,
or *dynamic*, i.e.:
	N_0 = can_queue/10 + 1,  (no devices yet)
	N_i = host::num_devices + N_(i-1)/2,  i > 0,

*please* suggest it, show it, and prove it. (See bottom of text.)

Just remember, the whole framework is one such that:

1. SCSI Core MUST NOT just blindly call the slab for more memory -- this *beats the whole purpose* of using the slab and we could just as well use kmalloc().

2. Adjustment should be done at scsi_put_command(). I.e. we're *holding off* commands from the slab on scsi_put_command(), depending on the heuristic. (Which by definition tells us something about the *future* and this is the whole point.) So that in the future we can be ok, should memory be scarce. I.e. do we need to decrease or increase the commands in free_list, depending on the current value of N_i.

In the example I gave above for a dynamic heuristic, it is easiliy shown that

	N_i = 2*(host::num_devices), i -> inf.

To show this is important as we don't want to drain the computer's memory.

The above heuristic would settle for N being 2 times the number of devices, which is good, since if can_queue = 255,
we have N_0 = 26, but after a few generations, we don't want to keep so many commands around, and given we have only one device, we settle to N_i = 2, i -> inf.

This was just an example, of course. You can model your work similarly.

BTW, if you are so ardent on this whole issue I can put code in for the heuristic, and you can supply N_0 and N_i.

Luben

Please do not reinvent the mm/mempool.c functionality.

'twould be better to just use it ;)

Unfortunately, in this instance, mempool is a slight overkill. The problem is that we need to guarantee that a command (or set of commands) be available to a given device regardless of what's going on in the rest of the system. Thus we might need a mempool for each active device, rather than a mempool for all devices and a mechanism for giving fine grained control to the pool depth per device. Mempool would fit all of the above, I was just concerned that it looks to be a rather heavy addition (in terms of structure size) per device.

James

A lot depends on the context of the allocation. Can the caller sleep? Is the caller using GFP_ATOMIC/__GFP_HIGH?

(What file-n-line should I be looking at, anyway?)

Bear in mind that on the swapout path, the calling process has PF_MEMALLOC set. This is a strong and successful mechanism - it allows the caller to dip into the final page reserves which are denied to even GFP_ATOMIC allocations. There's maybe a megabyte or two there.

Could be that there's no problem to be solved here. It depends on whether these allocations are occurring in process context or not.

It's 40-odd bytes, plus 4*max_reservation bytes. Fairly lean.

No (generally) - we are in our request function, and are called via soft irq and from anywhere that blk_run_queues is called. Calls outside of the request function can sleep (but they are not used for regular IO).

Yes normally GFP_ATOMIC

If you have bk, bk://linux-scsi.bkbits.net/scsi-combined-2.5, file is drivers/scsi/scsi_lib.c - the calls to scsi_getset_command. That tree also has other scsi changes, as well as changes related to the new allocation scheme.

In 2.5.59 this was a call to scsi_allocate_device. The current 2.5 allocation alogrithm used by scsi_allocate_device is poor. In the new code, scsi_allocate_device is replaced by a call to scsi_getset_command.

scsi_getset_command effectively calls kmem_cache_alloc(some_cache, flags).

My complaint that started the thread is that (generally) we don't have fairness across devices (scsi_device) on kmem_cache_alloc failure - a failure can put us in a timeout/poll like mode, but anyone with an IO already in flight can allocate (from kmem_cache_alloc or a single scsi_cmnd saved per host adapter) and issue another IO before the device that had a kmem_cache_alloc failure.

So, a swap disk could potentially be starved from issuing IO during low memory conditions.

But, there is the extra scsi_cmnd per host adapter (not per device), and it is always refilled (if empty) when a scsi_cmnd completes for that adapter, this combined with filling of the cache via continued IO use might prevent most (and hopefully all) such failures.

-- Patrick Mansfield

I think the case is that there is no problem to be solved. One command per host is enough to keep each host running, and that's enough to keep the system running. If we are ever low enough on mem that we get down to failing scsi command allocations, the system is already hurting. The complaint was that a device doing something other than swap could starve a swap device. I don't buy that. If the device is doing constant reads then it's going to run out of mem eventually and block just like our allocations are, if it's writing then it very likely is freeing up just as many pages as the swap operation would be. In short, I think if we keep the disk subsystem running, even if crippled with just one command, the problem becomes self correcting and there isn't much for us to solve. Of course, that's just my 5 minute analysis, someone feel free to prove me wrong.

  Doug Ledford <[email protected]>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606

I'd agree with that.

Plus there's the PF_MEMALLOC thing which gives swapper-outers an extra megabyte or two. If _any_ of these allocations are happening in process context then they will benefit from this.

If all the allocations are happening at interrupt/softirq time then some changes might be needed.

But I haven't been able to break it, using aix7xxx, with quite harsh testing.

I agree with the analysis: The system can make forward progress as long as we have only one guaranteed command.

However, I do worry about the performance under memory pressure. I don't think only having one command pre allocated per HBA is sufficient to ensure efficient swap out behaviour under load. The question, of course, is what do we need to do to make it more efficient?

Andrew, these patches are now in Linus' BK, so if you want to take a look and see how our loaded behaviour is, I'd be grateful.

James

Looks pretty straightforward. Most of the allocations are GFP_KERNEL, which is a good sign.

One could have designed it to support a pool of >1 command from the outset, but it's unlikely to be necessary.

(linux-scsi@vger doesn't send messages to or from oneself to or from oneself.
This upsets one's filing system. Does this irritate otherselves as much as
this self?)

Yes, I also that it's not likely to be necessary.

The functionality of support for more than one is there (i.e. in the freeing-list code). As I mentioned in this thread before, I had *no* idea (and still have none) on _what_ number to settle if greater than one.

OTOH, if the powers that be decide that more than one is nevertheless necessary, a mempool, I think, would be quite appropriate.

Luben

Doug,

I had argued some time ago for reserved and priority allocation for swap under block period. Regardless if this is scsi/ata/sas/sata the issue is fundamential. I spent a fair amount of time debating memory pressure against swap in combination of saturated device request queues, with Rik Riel. This is one layer above the LLDD and regardless if you reserve 1,2,N-1,N command slots in the queuedcommand list. If the request can not be obtained from block because all request slots are stuffed full, you have to deploy an out-of-bounds operation.

Progress will not happen period. Elevator sorting on swap is silly! If it is still done, you can not get there from here. Any swap-io run via the elevator must be inserted in the front of the request queue, period. This will provide you N+1 jump without fracturing all the commands in process or in flight. Since it would be priority, one can stuff it with the SPECIAL marker and boost it to the head of the queuedcommand list.

Maybe I am on crack, but so is the design delpoyed to date.

	(from top->bottom and not bottom->up)

Cheers,

Andre Hedrick
LAD Storage Consulting Group

In essence, all we really need to guarantee under memory pressure is that I/O which is being used to clear memory (i.e. for the swap device) will eventually proceed. This is the weakest necessary assumption for the system to make forward progress. Having a single command per device (or even just a single available command) guarantees this since if it is outstanding, it will eventually return and be re-used for clearing memory, which is all that is required.

With a single command per host, there is a starvation issue if you have heavy I/O to a device whose controller also contains the swap. However, it would have to be fairly pathological conditions to continue doing heavy I/O under memory pressure while starving the swap device.

Now we get into tuning: Even if the system is making forward progress, it might be doing it erratically, so how best do we ensure that the memory clearing I/O proceeds.

Right: a single command reserved per swap device would be sufficient to assure a steady stream of memory clearing I/O, which is probably sufficient for most purposes.

That's true, but again, it's a system tuning issue. The optimal thing to do for SCSI is to issue a new command for a device that just returned one because we know it has all the resources to hand.

What we do under memory pressure needs to be separated from what we do ordinarily.

This is theoretically possible, but unlikely: all of the allocated commands must eventually return. I can't think of any non pathological load scenarios where we can load up the command queues so completely from userland as to cause complete starvation of the swap devices...but doubtless somebody will come up with one.

James