I/O problems in 2.4.19/2.4.20/2.4.21-rc3

Hello all :-)

I have a problem since Linux Kernel 2.4.19. Copying huge amount of data gives me pauses where pauses are disk io pauses, keyboard does not accept input and mouse won't move. This depends, sometimes those pauses are 1 to 2 seconds, sometimes even more up to 15 seconds where I can not do anything with my linux but waiting :-(

For the past weeks I've searched google alot and found almost the same reports there but with imho no real fixes. I've tried many different kernel patches I've found while searching google (2.4-aa, 2.4-ck, 2.4-wolk, 2.4-rmap) but none of them fixes the problem I've experienced. The kernel patches 2.4-ck and 2.4-wolk are very good, those pauses are almost gone, but also the throughput is horribly decreased. Yesterday, mcp from #kernelnewbies told me, that this is the decrease of nr_requests to 32 (or maybe 4? I don't remember exactly). I've also tried 2.4-aa patch because I've read about his lowlatency elevator which should fix these pauses. Unfortunately the pauses are still there and also a decrease in throughput :-(

I've switched my desktop machine back from 2.4.20 to 2.4.18 because these pauses are really annoying me. I wonder what changes were made to 2.4.19 causing these pauses. Please don't get me wrong but it seems so that the Linux Kernel is not ready for desktop yet, and I even wonder about guaranteed io throughput for serverusage (please read down below)

This is also not a problem of my hardware. I've tried the same scenario on almost 20 different machines in my company, starting with a small 500mhz cpu and udma 100 intel controller up to a Pentium 4 with 2,4 GHz with an Adaptec U160 scsi controller and u160 scsi disks with software raid-0 and raid-1.

Carl-Daniel Hailfinger has been very helpfull yesterday on #kernelnewbies trying to track this behaviour down. He advised me to use SysRq-T and to press this key combination while there are io pauses. Maybe he will find out what's going on there :-)

Beside that, I've also noticed that there is no guaranteed io throughput while copying data in 2.4.18 up to 2.4.21-rc3. My machine has 512MB of memory and 512MB swap. Right after bootup of linux, there is guaranteed io throughput until the memory is almost completely used (with buffers or cache? ... /proc/meminfo tells me so) and linux starts to swap. After this, copying data starts up with 30mb per second and goes down real fast to 1mb per second and even more worse down to ~250kb per second, goes up to 10mb per second and so on, so this varies alot.

Anyway, I've also read about kernel 2.5 and that this kernel should fix all of the above I've mentioned. So by reading all these great oppinions about kernel 2.5 I've tried it out last week and I just have to say that I cannot see any advantages, at least not for these 2 cases I've mentioned :-(((

Is it just me or are there many others noticing this too?

Please excuse my bad english but I hope everyone understands me :-)

PS: Should I CC this to Marcello Tosatti and Linus Torvalds too? I haven't done this yet but maybe it may help because both are the maintainers of 2.4/ 2.5 (at least that's what I've found in google). Sorry, I am using Linux since ~ 1 1/2 years now and my knowledge about the Linux Kernel is not that big.

Thank you so much and have a nice weekend :-)

bye, Chris

On Friday 23 May 2003 15:00, Christian Klose wrote:

Hello all again :-)

I forgot to mention that this is filesystem independant. ext2, ext3, reiserfs; always same problem.

bye, Chris

On Friday 23 May 2003 15:46, Christian Klose wrote:

Moin Christian,

it _seems_ the offending diff is the attached one. Went in into .19-pre5.

More to come.

ciao, Marc

diff -urN linux-2.4.19-pre4/drivers/block/ll_rw_blk.c linux-2.4.19-pre5/drivers/block/ll_rw_blk.c
@@ -117,16 +117,6 @@
  */
 int * max_sectors[MAX_BLKDEV];
 
-/*
- * The total number of requests in each queue.
- */
-static int queue_nr_requests;
-
-/*
- * The threshold around which we make wakeup decisions
- */
-static int batch_requests;
-
 static inline int get_max_sectors(kdev_t dev)
 {
 	if (!max_sectors[MAJOR(dev)])

@@ -180,7 +170,7 @@
  **/
 void blk_cleanup_queue(request_queue_t * q)
 {
-	int count = queue_nr_requests;
+	int count = q->nr_requests;
 
 	count -= __blk_cleanup_queue(&q->rq[READ]);
 	count -= __blk_cleanup_queue(&q->rq[WRITE]);

@@ -330,31 +320,64 @@
 	spin_unlock_irqrestore(&io_request_lock, flags);
 }
 
+/** blk_grow_request_list
+ *  @q: The &request_queue_t
+ *  @nr_requests: how many requests are desired
+ *
+ * More free requests are added to the queue's free lists, bringing
+ * the total number of requests to @nr_requests.
+ *
+ * The requests are added equally to the request queue's read
+ * and write freelists.
+ *
+ * This function can sleep.
+ *
+ * Returns the (new) number of requests which the queue has available.
+ */
+int blk_grow_request_list(request_queue_t *q, int nr_requests)
+{
+	spin_lock_irq(&io_request_lock);
+	while (q->nr_requests < nr_requests) {
+		struct request *rq;
+		int rw;
+
+		spin_unlock_irq(&io_request_lock);
+		rq = kmem_cache_alloc(request_cachep, SLAB_KERNEL);
+		spin_lock_irq(&io_request_lock);
+		if (rq == NULL)
+			break;
+		memset(rq, 0, sizeof(*rq));
+		rq->rq_status = RQ_INACTIVE;
+		rw = q->nr_requests & 1;
+		list_add(&rq->queue, &q->rq[rw].free);
+		q->rq[rw].count++;
+		q->nr_requests++;
+	}
+	q->batch_requests = q->nr_requests / 4;
+	if (q->batch_requests > 32)
+		q->batch_requests = 32;
+	spin_unlock_irq(&io_request_lock);
+	return q->nr_requests;
+}
+
 static void blk_init_free_list(request_queue_t *q)
 {
-	struct request *rq;
-	int i;
+	struct sysinfo si;
+	int megs;		/* Total memory, in megabytes */
+	int nr_requests;
 
 	INIT_LIST_HEAD(&q->rq[READ].free);
 	INIT_LIST_HEAD(&q->rq[WRITE].free);
 	q->rq[READ].count = 0;
 	q->rq[WRITE].count = 0;
+	q->nr_requests = 0;
 
-	/*
-	 * Divide requests in half between read and write
-	 */
-	for (i = 0; i < queue_nr_requests; i++) {
-		rq = kmem_cache_alloc(request_cachep, SLAB_KERNEL);
-		if (rq == NULL) {
-			/* We'll get a `leaked requests' message from blk_cleanup_queue */
-			printk(KERN_EMERG "blk_init_free_list: error allocating requests\n");
-			break;
-		}
-		memset(rq, 0, sizeof(struct request));
-		rq->rq_status = RQ_INACTIVE;
-		list_add(&rq->queue, &q->rq[i&1].free);
-		q->rq[i&1].count++;
-	}
+	si_meminfo(&si);
+	megs = si.totalram >> (20 - PAGE_SHIFT);
+	nr_requests = 128;
+	if (megs < 32)
+		nr_requests /= 2;
+	blk_grow_request_list(q, nr_requests);
 
 	init_waitqueue_head(&q->wait_for_requests[0]);
 	init_waitqueue_head(&q->wait_for_requests[1]);

@@ -610,7 +633,7 @@
 	 */
 	if (q) {
 		list_add(&req->queue, &q->rq[rw].free);
-		if (++q->rq[rw].count >= batch_requests &&
+		if (++q->rq[rw].count >= q->batch_requests &&
 				waitqueue_active(&q->wait_for_requests[rw]))
 			wake_up(&q->wait_for_requests[rw]);
 	}

@@ -802,7 +825,7 @@
 		 * See description above __get_request_wait()
 		 */
 		if (rw_ahead) {
-			if (q->rq[rw].count < batch_requests) {
+			if (q->rq[rw].count < q->batch_requests) {
 				spin_unlock_irq(&io_request_lock);
 				goto end_io;
 			}

@@ -958,6 +981,7 @@
 		BUG();
 
 	set_bit(BH_Req, &bh->b_state);
+	set_bit(BH_Launder, &bh->b_state);
 
 	/*
 	 * First step, 'identity mapping' - RAID or LVM might

@@ -1149,12 +1173,9 @@
 	blkdev_release_request(req);
 }
 
-#define MB(kb)	((kb) << 10)
-
 int __init blk_dev_init(void)
 {
 	struct blk_dev_struct *dev;
-	int total_ram;
 
 	request_cachep = kmem_cache_create("blkdev_requests",
 					   sizeof(struct request),

@@ -1170,22 +1191,6 @@
 	memset(max_readahead, 0, sizeof(max_readahead));
 	memset(max_sectors, 0, sizeof(max_sectors));
 
-	total_ram = nr_free_pages() << (PAGE_SHIFT - 10);
-
-	/*
-	 * Free request slots per queue.
-	 * (Half for reads, half for writes)
-	 */
-	queue_nr_requests = 64;
-	if (total_ram > MB(32))
-		queue_nr_requests = 128;
-
-	/*
-	 * Batch frees according to queue length
-	 */
-	batch_requests = queue_nr_requests/4;
-	printk("block: %d slots per queue, batch=%d\n", queue_nr_requests, batch_requests);
-
 #ifdef CONFIG_AMIGA_Z2RAM
 	z2_init();
 #endif

@@ -1296,6 +1301,7 @@
 EXPORT_SYMBOL(io_request_lock);
 EXPORT_SYMBOL(end_that_request_first);
 EXPORT_SYMBOL(end_that_request_last);
+EXPORT_SYMBOL(blk_grow_request_list);
 EXPORT_SYMBOL(blk_init_queue);
 EXPORT_SYMBOL(blk_get_queue);
 EXPORT_SYMBOL(blk_cleanup_queue);

On Friday 23 May 2003 15:46, Christian Klose wrote:

Hi Christian,

You've mentioned that you also tried 2.5 and you don't see any difference there. I am very interrested to hear what version of 2.5 you tried.

I am using 2.5.69 + some patches and it does not suck or behave brain dead like 2.4 does. For me I've quit with 2.4 from now on.

2.5 still has many things to do / to fix but 2.5 is _active_ developed and fixes _are_ going into 2.5 where 2.4 does not accept real bug fixes.

If you want to give 2.5.69 a try, please apply the attached patch if you want to use it for your desktop. It gives you more interactivity.

ciao, Marc

@@ -65,7 +65,7 @@
  * they expire.
  */
 #define MIN_TIMESLICE		( 10 * HZ / 1000)
-#define MAX_TIMESLICE		(200 * HZ / 1000)
+#define MAX_TIMESLICE		( 10 * HZ / 1000)
 #define CHILD_PENALTY		50
 #define PARENT_PENALTY		100
 #define EXIT_WEIGHT		3

On Saturday 24 May 2003 16:28, William Lee Irwin III wrote:

Hi wli,

I somewhat agree with you but this "properly described" are all the bug reports on lkml containing "bad interactivity in 2.5, cpu starving in 2.5" and such...

This isn't a shenanigan, at least not for the interactivity for a desktop. This is a workaround for users who are complaining about bad interactivity in 2.5!

ciao, Marc

Hi Christian,

err, could you please tell me why you are writing "ciao, Marc" like I do in my emails if you are pretending beeing Christian Klose??

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Even though you're not Marc I do agree with you. The problem is well described as either poor interactivity (the window wiggle test) or starvation in the presence of certain scheduler hogs (for whatever reason) since the interactivity patch from mingo. Dropping the max timeslice is a bandaid but destroys priority based timeslice scheduling. Dropping the min timeslice will bring this back, but at some point the timeslice will be so low that low priority cpu intensive tasks will spend most of their time cache trashing.

A reasonable compromise for the desktop would be
min 5
max 25
but some granularity will be lost in the different sizes of timeslices at different priorities.

is there any point having longer timeslices than this?

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQE+0Bv6F6dfvkL3i1gRAhpfAKCG3fjkK02lYbAs1p3978rSL/PYAQCcCeK7 gHqR6bgrITE3CSjKCqntw+g=
=rq1o
-----END PGP SIGNATURE-----

The fact that it's a "bandaid" and that it "destroys priority-based timeslice scheduling" makes it a shenanigan. If you're having problems solved by capping timeslices, you have someone's timeslice and/or priority growing too large for some reason.

It'd be far better to help figure out what went wrong.

-- wli

I don't disagree it's a shenanigan.

So there is a benefit to timeslices being as large as 200ms? I'll take your word for it.

Love to help. No idea where to begin. All we can do is report what helps the symptoms and hope those in the know can decipher it from that.

Con