Skip to contentSkip to conversation

Poor performance during disk writes

DISCUSSION Dieter Nützel 6 participants 7 messages Dec 20, 2001 – Dec 21, 2001 Block layer

lore
Share & tools
lore
#0 lore raw
Quoted 1 line · On Thursday, 20.12.201, 10:49 Helge Hafting wrote:

jlm wrote:

[-]

Quoted 7 lines
Quoted 3 lines

So I guess I don't really care what mode the hard drive is operating in (udma, mdma, dma or plain ide), I just don't want to have to go get a cup of coffee while the hard drive saves some data. Is there a "don't

Devices generally get the cpu before anything else. A good disk system don't need much cpu. Running IDE in PIO mode require a lot
of cpu though. Using any of the DMA modes avoids that.

Amen..
Sorry, Helge sure you are right in theory but try dbench 32 (maybe bonnie/bonnie++) and playing an MP3/Ogg-Vorbis in parallel... That's my first test on any "new" kernel version.

Even with an 1 GHz Athlon II, 640 MB, U160 DDYS 18 GB, 10k IBM disk (on an AHA-2940UW) it stutters like mad. I am running all my kernel _with_ Robert Love's preempt + lock-break patches and it doesn't solve the problem. CPU load is (very) low but it do not work like it should.

Quoted 4 lines
Quoted 2 lines

pre-empt the rest of the system" switch for the eide drives? Is there something fundamental/unique going on here that I'm missing?

dma, udma, etc. is that switch. It lets the cpu do other work (such as redrawing X) while the disk is busy. Plain ide is what you don't want.

See above the whole system show some bad hiccup.

Quoted 6 lines

The problem of waiting for other files or swapping while a really big write is going on is different. Get more drives, so the big writes go to one drive while you get stuff swapped in (or other file access) on other drive(s). The kernel is capable of getting fast response from one drive while another is completely bogged down with enormous writes.

Tried this already. Neither I put my test files (MP3/Ogg-Vorbis) in /dev/shm or a nother disk it do not change anything.

There must be something in the VFS?

-Dieter

Signature · 4 lines
Dieter Nützel
Graduate Student, Computer Science
University of Hamburg
@home: [email protected]
#1 lore raw
Quoted 16 lines · Dieter Nützel
Quoted 1 line · On Thursday, 20.12.201, 10:49 Helge Hafting wrote:

jlm wrote:

[-]

Quoted 7 lines
Quoted 3 lines

So I guess I don't really care what mode the hard drive is operating in (udma, mdma, dma or plain ide), I just don't want to have to go get a cup of coffee while the hard drive saves some data. Is there a "don't

Devices generally get the cpu before anything else. A good disk system don't need much cpu. Running IDE in PIO mode require a lot
of cpu though. Using any of the DMA modes avoids that.

Amen..
Sorry, Helge sure you are right in theory but try dbench 32 (maybe bonnie/bonnie++) and playing an MP3/Ogg-Vorbis in parallel... That's my first test on any "new" kernel version.

Quoted 4 lines

Even with an 1 GHz Athlon II, 640 MB, U160 DDYS 18 GB, 10k IBM disk (on an AHA-2940UW) it stutters like mad. I am running all my kernel _with_ Robert Love's preempt + lock-break patches and it doesn't solve the problem. CPU load is (very) low but it do not work like it should.

try it with vanilla 2.4.17-rc1. I just did and i'm getting no
stuttering at all. nice -n 20 dbench 32. Worked quite nicely. Of
course i'm using an ext3 fs which is more important than your cpu speed
or ram. This kind of discussion has been talked about and argued over
many times in the past here already. Too many factors go into this and in the end, dbench is _Meant_ to preempt everything else. if you want a real test find a real program you really use and use it.

Quoted 32 lines
Quoted 4 lines
Quoted 2 lines

pre-empt the rest of the system" switch for the eide drives? Is there something fundamental/unique going on here that I'm missing?

dma, udma, etc. is that switch. It lets the cpu do other work (such as redrawing X) while the disk is busy. Plain ide is what you don't want.

See above the whole system show some bad hiccup.

Quoted 6 lines

The problem of waiting for other files or swapping while a really big write is going on is different. Get more drives, so the big writes go to one drive while you get stuff swapped in (or other file access) on other drive(s). The kernel is capable of getting fast response from one drive while another is completely bogged down with enormous writes.

Tried this already. Neither I put my test files (MP3/Ogg-Vorbis) in /dev/shm or a nother disk it do not change anything.

There must be something in the VFS?

-Dieter

Signature · 9 lines
Dieter Nützel
Graduate Student, Computer Science
University of Hamburg
@home: [email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#2 lore raw
In reply to Dieter Nützel
Quoted 9 lines · Dieter N?tzel

Amen..
Sorry, Helge sure you are right in theory but try dbench 32 (maybe bonnie/bonnie++) and playing an MP3/Ogg-Vorbis in parallel... That's my first test on any "new" kernel version.

Even with an 1 GHz Athlon II, 640 MB, U160 DDYS 18 GB, 10k IBM disk (on an AHA-2940UW) it stutters like mad. I am running all my kernel _with_ Robert Love's preempt + lock-break patches and it doesn't solve the problem. CPU load is (very) low but it do not work like it should.

I tried this on my 600MHz Athlon with 768MB of RAM and U160 DDYS 36GB 10Krpm IBM disk on a Adaptect 39160 I managed to get it not to stutter at all. I was also using preempt + lockbreak and a few others. The crucial patch appeared to be from Andrew Morton and it involved tuning the elevator to avoid read starvation. A significantly helpful hardware suggestion regarding the sound card and drivers came from Linus himself, though.

Linus and others pointed out that applications are able to cause some drivers to generate a large number of interrupts by using small buffers and unfriendly ioctl's, especially esd. My workaround was to change out sound hardware and disable esd. If this is happening to you, /proc/profile should show handle_IRQ_event() and schedule() very high up. On the other hand, this shows up as a steady drain on system resources and excessive system time, not stuttering or skipping.

Andrew, I don't have the URL for that still floating around. Can you point Dieter to it?

Cheers,
Bill

#3 lore raw
In reply to Dieter Nützel
Quoted 3 lines · William Lee Irwin III

Andrew, I don't have the URL for that still floating around. Can you point Dieter to it?

It's here.

You need to run

elvtune -b N /dev/hdXX

where N=0 is "disable", N=1 is minimum read latency, N=6 is a reasonable setting.

drivers/block/elevator.c +56 -4
@@ -74,11 +74,10 @@ inline int bh_rq_in_between(struct buffe
 	return 0;
 }
 
-
 int elevator_linus_merge(request_queue_t *q, struct request **req,
 			 struct list_head * head,
 			 struct buffer_head *bh, int rw,
-			 int max_sectors)
+			 int max_sectors, int max_bomb_segments)
 {
 	struct list_head *entry = &q->queue_head;
 	unsigned int count = bh->b_size >> 9, ret = ELEVATOR_NO_MERGE;
@@ -116,6 +115,56 @@ int elevator_linus_merge(request_queue_t
 		}
 	}
 
+	/*
+	 * If we failed to merge a read anywhere in the request
+	 * queue, we really don't want to place it at the end
+	 * of the list, behind lots of writes.  So place it near
+	 * the front.
+	 *
+	 * We don't want to place it in front of _all_ writes: that
+	 * would create lots of seeking, and isn't tunable.
+	 * We try to avoid promoting this read in front of existing
+	 * reads.
+	 *
+	 * max_bomb_sectors becomes the maximum number of write
+	 * requests which we allow to remain in place in front of
+	 * a newly introduced read.  We weight things a little bit,
+	 * so large writes are more expensive than small ones, but it's
+	 * requests which count, not sectors.
+	 */
+	if (max_bomb_segments && rw == READ && ret == ELEVATOR_NO_MERGE) {
+		int cur_latency = 0;
+		struct request * const cur_request = *req;
+
+		entry = head->next;
+		while (entry != &q->queue_head) {
+			struct request *__rq;
+
+			if (entry == &q->queue_head)
+				BUG();
+			if (entry == q->queue_head.next &&
+					q->head_active && !q->plugged)
+				BUG();
+			__rq = blkdev_entry_to_request(entry);
+
+			if (__rq == cur_request) {
+				/*
+				 * This is where the old algorithm placed it.
+				 * There's no point pushing it further back,
+				 * so leave it here, in sorted order.
+				 */
+				break;
+			}
+			if (__rq->cmd == WRITE) {
+				cur_latency += 1 + __rq->nr_sectors / 64;
+				if (cur_latency >= max_bomb_segments) {
+					*req = __rq;
+					break;
+				}
+			}
+			entry = entry->next;
+		}
+	}
 	return ret;
 }
 
@@ -144,7 +193,7 @@ void elevator_linus_merge_req(struct req
 int elevator_noop_merge(request_queue_t *q, struct request **req,
 			struct list_head * head,
 			struct buffer_head *bh, int rw,
-			int max_sectors)
+			int max_sectors, int max_bomb_segments)
 {
 	struct list_head *entry;
 	unsigned int count = bh->b_size >> 9;
@@ -188,7 +237,7 @@ int blkelvget_ioctl(elevator_t * elevato
 	output.queue_ID			= elevator->queue_ID;
 	output.read_latency		= elevator->read_latency;
 	output.write_latency		= elevator->write_latency;
-	output.max_bomb_segments	= 0;
+	output.max_bomb_segments	= elevator->max_bomb_segments;
 
 	if (copy_to_user(arg, &output, sizeof(blkelv_ioctl_arg_t)))
 		return -EFAULT;
@@ -207,9 +256,12 @@ int blkelvset_ioctl(elevator_t * elevato
 		return -EINVAL;
 	if (input.write_latency < 0)
 		return -EINVAL;
+	if (input.max_bomb_segments < 0)
+		return -EINVAL;
 
 	elevator->read_latency		= input.read_latency;
 	elevator->write_latency		= input.write_latency;
+	elevator->max_bomb_segments	= input.max_bomb_segments;
 	return 0;
 }
 
drivers/block/ll_rw_blk.c +2 -1
@@ -690,7 +690,8 @@ again:
 	} else if (q->head_active && !q->plugged)
 		head = head->next;
 
-	el_ret = elevator->elevator_merge_fn(q, &req, head, bh, rw,max_sectors);
+	el_ret = elevator->elevator_merge_fn(q, &req, head, bh,
+				rw, max_sectors, elevator->max_bomb_segments);
 	switch (el_ret) {
 
 		case ELEVATOR_BACK_MERGE:
include/linux/elevator.h +13 -27
@@ -5,8 +5,9 @@ typedef void (elevator_fn) (struct reque
 			    struct list_head *,
 			    struct list_head *, int);
 
-typedef int (elevator_merge_fn) (request_queue_t *, struct request **, struct list_head *,
-				 struct buffer_head *, int, int);
+typedef int (elevator_merge_fn)(request_queue_t *, struct request **,
+				struct list_head *, struct buffer_head *bh,
+				int rw, int max_sectors, int max_bomb_segments);
 
 typedef void (elevator_merge_cleanup_fn) (request_queue_t *, struct request *, int);
 
@@ -16,6 +17,7 @@ struct elevator_s
 {
 	int read_latency;
 	int write_latency;
+	int max_bomb_segments;
 
 	elevator_merge_fn *elevator_merge_fn;
 	elevator_merge_cleanup_fn *elevator_merge_cleanup_fn;
@@ -24,13 +26,13 @@ struct elevator_s
 	unsigned int queue_ID;
 };
 
-int elevator_noop_merge(request_queue_t *, struct request **, struct list_head *, struct buffer_head *, int, int);
-void elevator_noop_merge_cleanup(request_queue_t *, struct request *, int);
-void elevator_noop_merge_req(struct request *, struct request *);
-
-int elevator_linus_merge(request_queue_t *, struct request **, struct list_head *, struct buffer_head *, int, int);
-void elevator_linus_merge_cleanup(request_queue_t *, struct request *, int);
-void elevator_linus_merge_req(struct request *, struct request *);
+elevator_merge_fn elevator_noop_merge;
+elevator_merge_cleanup_fn elevator_noop_merge_cleanup;
+elevator_merge_req_fn elevator_noop_merge_req;
+
+elevator_merge_fn elevator_linus_merge;
+elevator_merge_cleanup_fn elevator_linus_merge_cleanup;
+elevator_merge_req_fn elevator_linus_merge_req;
 
 typedef struct blkelv_ioctl_arg_s {
 	int queue_ID;
@@ -54,22 +56,6 @@ extern void elevator_init(elevator_t *, 
 #define ELEVATOR_FRONT_MERGE	1
 #define ELEVATOR_BACK_MERGE	2
 
-/*
- * This is used in the elevator algorithm.  We don't prioritise reads
- * over writes any more --- although reads are more time-critical than
- * writes, by treating them equally we increase filesystem throughput.
- * This turns out to give better overall performance.  -- sct
- */
-#define IN_ORDER(s1,s2)				\
-	((((s1)->rq_dev == (s2)->rq_dev &&	\
-	   (s1)->sector < (s2)->sector)) ||	\
-	 (s1)->rq_dev < (s2)->rq_dev)
-
-#define BHRQ_IN_ORDER(bh, rq)			\
-	((((bh)->b_rdev == (rq)->rq_dev &&	\
-	   (bh)->b_rsector < (rq)->sector)) ||	\
-	 (bh)->b_rdev < (rq)->rq_dev)
-
 static inline int elevator_request_latency(elevator_t * elevator, int rw)
 {
 	int latency;
@@ -85,7 +71,7 @@ static inline int elevator_request_laten
 ((elevator_t) {								\
 	0,				/* read_latency */		\
 	0,				/* write_latency */		\
-									\
+	0,				/* max_bomb_segments */		\
 	elevator_noop_merge,		/* elevator_merge_fn */		\
 	elevator_noop_merge_cleanup,	/* elevator_merge_cleanup_fn */	\
 	elevator_noop_merge_req,	/* elevator_merge_req_fn */	\
@@ -95,7 +81,7 @@ static inline int elevator_request_laten
 ((elevator_t) {								\
 	8192,				/* read passovers */		\
 	16384,				/* write passovers */		\
-									\
+	0,				/* max_bomb_segments */		\
 	elevator_linus_merge,		/* elevator_merge_fn */		\
 	elevator_linus_merge_cleanup,	/* elevator_merge_cleanup_fn */	\
 	elevator_linus_merge_req,	/* elevator_merge_req_fn */	\
#4 lore raw
Quoted 4 lines · Andrew Morton

You need to run
elvtune -b N /dev/hdXX
where N=0 is "disable", N=1 is minimum read latency, N=6 is a reasonable setting.

I'm curious, why was max_bomb_segments dropped the last time it was in the tree ? I recall it happening, but the reason escapes me.

Dave.

Signature · 2 lines
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs
#5 lore raw
In reply to Andrew Morton
Quoted 10 lines · Dave Jones
Quoted 4 lines · Andrew Morton

You need to run
elvtune -b N /dev/hdXX
where N=0 is "disable", N=1 is minimum read latency, N=6 is a reasonable setting.

I'm curious, why was max_bomb_segments dropped the last time it was in the tree ? I recall it happening, but the reason escapes me.

Fooled me too the first time, read Andrew's patch though. It isn't related at all.

Signature · 1 line
Jens Axboe
#6 lore raw
In reply to Dieter Nützel

drivers/block/ll_rw_blk.c again: · line 694

Quoted 9 lines · Andrew Morton
drivers/block/ll_rw_blk.c +2 -1
@@ -690,7 +690,8 @@ again:
 	} else if (q->head_active && !q->plugged)
 		head = head->next;
 
-	el_ret = elevator->elevator_merge_fn(q, &req, head, bh, rw,max_sectors);
+	el_ret = elevator->elevator_merge_fn(q, &req, head, bh,
+				rw, max_sectors, elevator->max_bomb_segments);

merge function can just grab max_bomb_segments ala

	int mbs = q->elevator.max_bomb_segments

so no need to modify the merge functions.

Signature · 1 line
Jens Axboe