Feature proposal (scheduling related)

With the current scheduler we can prioritize the CPU usage for each process. What I think would be extremely useful (as I have needed it many times) is the scheduling of disk I/O and net I/O traffic. 2 examples showing the importance (the numbers are estimations just to explain whati I mean):

1)I 'm connected to the internet via dial-up, therefore I only have 40 kbits of bandwidth available. What I want to do is listen to icecast radio via xmms (at 22 kbits), download the kernel sources with wget, and browse the web at the same time. Currently I think that this is *impossible* (correct me if I'm wrong) as the radio will be full of pauses and the browsing experience painfully slow. What I would like to be able to do (let's suppose nice has the --net option to set net I/O priority):

$ nice --net -1 xmms
$ nice --net 1 wget ftp://.../KernelSources.tar.bz2
$ mozilla
This way, xmms which has top priority whould always get the 22kbits it needs. What remains should go to the browser when I ask for a web page, and when the browser doesn't request anything (let's say I'm reading a big doc in tldp) what remains should go to wget. Wget has lower priority and won't irritate the browsing experience, though the file will be downloaded when there is free bandwidth.

2) Normally mozilla starts in 5 seconds after intense disk I/O to load all needed libraries. If I run in the background a long disk intense process
(like find / -name 'whatever' -xdev) loading mozilla could need 20 boring seconds, or doing other simple tasks might be irritating slow. What I would like to be able to do is (once again let's suppose nice has the --disk option to set disk I/O priority):

$ nice --disk 1 find / -name 'whatever' -xdev
$ mozilla
and load mozilla ,which has the default disk priority 0, fast. The scheduler should give to mozilla most disk troughput when it needs it.

Notes:
1) PLEASE CC REPLIES BACK TO ME since I 'm not subscribed to the list (I can't stand the traffic). However I 'll be checking periodically the list via NNTP.
2) As I have no idea of kernel programming I hope what I propose is aplicable and relevant to the kernel, as I believe. Sorry if not.
3) I hope what I propose is implementable using the existing scheduler. It would be nice to have one scheduler to handle them all.
4) I believe that these features don't need a lot of CPU power because the disk and net I/O troughput are relatively slow.
5) If you think that UNIX tradition forbits what I propose I must say that these features could be invisible to programs, setting the new priorities to the default 0 or maybe to the same number as CPU priority.

Thank you very much for your time,
Dimitris

I think what you need to look at is network-level QoS, this is far too intrusive for what you are trying to do. Try tldp.org.

David.

Basically, you're stuck. The biggest part of the problem is that although you can certainly control the outbound packets, you have no real control over when inbound packets arrive at the other end of your dial-up. One person suggested using QoS to help things along - but that needs to be implemented at the OTHER end of the dial-up - which means unless your provider does QoS on the terminal server, you're basically stuck. Packets will probably just get queued up in order of arrival.

And even if the packets do get prioritized so your xmms traffic "goes first", that's still not going to be perfect, due to a phenomenon canned 'jitter', which is a measure of how "bursty" the traffic is, and how much *variability* there is.

Let's say you have xmms going, and a very slow wget. The last 5 packets of xmms traffic arrived when the modem was idle, and immediately start coming across, so the jitter is near zero. Now a 6th xmms packet arrives - but the modem is 3 bytes into transmitting a 1500 byte wget packet. Wham! that packet gets to wait 300ms for its turn, and you just got 300ms of 'jitter'. If the xmms didn't have 300ms of sound already buffered, you're screwed and about to have a drop-out. QoS doesn't help here - the best it can do is guarantee that you only have to wait for the currently transmitting packet and not for any already-queued packets. (As an aside - TCP windowing will probably guarantee that any given connection will only have 1 or 2 packets queued up - but if you're going to run 5 concurrent data-transfer connections, that's still 5-10 packets. If you don't have QoS, any given packet *could* end up with several seconds of latency here, and the exact order of arrival could make your jitter anywhere from 0 to several seconds...

You *might* be able to improve the jitter characteristics by dropping the MTU on the dialup connection from the 1500 byte range down to 576, but this has its own problems. First, you're going to be sending 3x as many packets, so the overhead of packet headers is going to triple, cutting into your effective bandwidth. Second, there are often issues with windowing, fragmented packets, and PMTU Discovery when you do this.

Again, here there be serious and nasty dragons. The problem is that you have to balance several competing issues:

1) Trying to maintain good overall disk throughput - if you keep pushing Mozilla's disk requests *right now*, regardless of what the disk is doing, the disk ends up spending a LOT of time just moving the heads back and forth. Even at 5ms a shot, seeks are *expensive*, so you often want to re-order the requests to minimize the distance travelled. Think of it as the difference between planning your trip to the grocery store, the gas station, the post office, and the bank to make it shortest, as opposed to going back home after each one because there's something at home that needs doing Right Now...

2) There's a phenomenon called "starvation". See that 'find' command in your example? If mozilla is disk-hungry enough, bad I/O scheduling can mean that 'find' command will sit there *forever*, tying up resources the whole time. This can cause issues. For instance - if you've flagged 'mozilla' as the process that gets first shot at the disk, what do you do if you start paging to the swap area, and some OTHER process has to read a page in? What if that "other process" is the X server or your window manager? Think REALLY hard here
- just saying "I'll renice them too" is NOT the right answer.. .;)

There are a few things that help in the general real world but not mathematical sense. Use an ftp client like gnome-ftp which can set the rate it accepts data and window sizes. It'll still jam the modem a little when it starts a transfer but then it'll generally be ok if you have a bit of buffering for your icecast stream.

More info. I use a 56k dialup link and PPD to essentially become my own ISP for my network at home. The machines at home are nodes on the company's LAN. I have two dedicated machines, one at work and one at home who's only purpose is to forward IP packets. These machines are, otherwise, idle.

When I am using ftp to download some work to one of my other home computers, it is functional impossible to do any useful work on any other connection. The connections persist, but no data will get through when a FTP transfer is in progress except for the FTP data. Somehow the FTP data stream is able to hog the entire transmission bandwidth. Attempts to "fix" it be tuning Nagle off, Van Jacobson, etc., all off doesn't do anything useful.

I have from time-to-time reported as far back as Linux 2.0, the fact that there are TCP stalls when using PPP. These stalls still persist with 2.4.20 and I've just had to accept the so- called "fact" that 1 to 10 second stalls when using PPP are "normal".

At one time I thought it was just that the modems had lost sync so I set up a wire-to-wire PPP link here at work. The stalls still exist when using the RS-232C link with a null- modem cable. Evidence is that packets are lost, even with a direct connection. It takes <too much> time for Linux to discover the missing datagram and have it re-sent. When a stream of data (FTP) is being sent, nothing seems to get lost except the ACKs from the receiving end. It seems as though RS-232C is not full duplex. If something is being sent, nothing will be received and vice versa.

But... If you write to RS-232C with a test program, with pins 2,3 shorted (loop-back), you can always read what you wrote with no errors IFF hardware CRTCTS is turned OFF.

If I enable hardware handshaking, necessary for a modem, with RTS jumpered to CTS, all bets are off. There are many missing bytes, even at 9600 baud. However, when connected through modems, this affect goes away. So, it seems as though the PPP problem is related to RS-232C problems, but I'll be damned if I can find it.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

You need QoS at the router level to resolve this. Since you are running your own routers to connect your ethernet segments, QoS should be done at both ends of the connection. If it's available on your distro, try wondershaper, it's a nice script which you tell your upstream and downstream rates and then it adjusts QoS parameters to provide great response. The most important thing is that it prioritises ACK packets above everything else. This helps a lot when there is heavy traffic (FTP for example) in both directions at the same time.

Greets, Antonio.

Alternately, openbsd can do a similar thing with great results (although I haven't tried it over serial/pppd.) And it works reasonably well even when done on only one end.

http://www.benzedrine.cx/ackpri.html

Disconnect <[email protected]>

Wondershaper usually does too.

It's all the same stuff.

I'm sure it's a serious issue, and yet my network has QoS set up and the low priority flows still eventually get through just fine even though it has much lower bandwidth available than my disk controller.

I think it would be really cool to be able to control disk i/o with the same level of flexibility as network i/o. I could see setting up cbq trees that can key off things like whether it's paging, a blocking/nonblocking i/o, or a nonblocking i/o. They could also see what user owns the process, and what inode the process's executable image is.

I would just wonder about the overhead.

greg

But the incoming packets are those that I request to be sent to me (most times). When I know the packet size I will accept I may not request all packets now, but a bit later, to leave bandwidth for other requested packets of higher priority.

This is what I don't like, but I bet there must be a way to slow down the other end, even if it has not QoS.

What might help, and needs only be implemented on my side of the connection, is requesting the packets of higher priorities first. And if I know my total bandwidth, perhaps I can request as many packets needed to fill it (and not flood it like it happens now).

Thank you all for your attention,
Dimitris

Yes, this is exactly what wondershaper does now.

Please give it a try.

I use it on a dsl line, and it works wonders. (pardon the pun).

Take a look at http://trash.net/~kaber/imq/
Does wonders here, both local and forwarded traffic.

Cheers,
André Tomt
[email protected]

Hi!

Yes that would be nice, and in 2.5 timeframe there was patch doing that. Port it to 2.6 an test it!

				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

Pavel,

Do you remember who wrote those or where one can find it?

Hi!

After a bit of searching I found this. I'm not 100% sure this is the same one I am remembering, but it looks like that.

http://marc.theaimsgroup.com/?l=linux-kernel&m=103962160319984&w=2

[No, I'm probably remembering some other patch, this looks way too simple, but may be good point to start...]
Pavel

When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]