Poor performance during disk writes
Quoted 1 line · On Thursday, 20.12.201, 10:49 Helge Hafting wrote:
jlm wrote:
[-]
Quoted 7 lines
Quoted 3 lines
So I guess I don't really care what mode the hard drive is operating in (udma, mdma, dma or plain ide), I just don't want to have to go get a cup of coffee while the hard drive saves some data. Is there a "don't
Devices generally get the cpu before anything else. A good disk system don't need much cpu. Running IDE in PIO mode require a lot
of cpu though. Using any of the DMA modes avoids that.
Amen..
Sorry, Helge sure you are right in theory but try dbench 32 (maybe bonnie/bonnie++) and playing an MP3/Ogg-Vorbis in parallel... That's my first test on any "new" kernel version.
Even with an 1 GHz Athlon II, 640 MB, U160 DDYS 18 GB, 10k IBM disk (on an AHA-2940UW) it stutters like mad. I am running all my kernel _with_ Robert Love's preempt + lock-break patches and it doesn't solve the problem. CPU load is (very) low but it do not work like it should.
Quoted 4 lines
Quoted 2 lines
pre-empt the rest of the system" switch for the eide drives? Is there something fundamental/unique going on here that I'm missing?
dma, udma, etc. is that switch. It lets the cpu do other work (such as redrawing X) while the disk is busy. Plain ide is what you don't want.
See above the whole system show some bad hiccup.
Quoted 6 lines
The problem of waiting for other files or swapping while a really big write is going on is different. Get more drives, so the big writes go to one drive while you get stuff swapped in (or other file access) on other drive(s). The kernel is capable of getting fast response from one drive while another is completely bogged down with enormous writes.
Tried this already. Neither I put my test files (MP3/Ogg-Vorbis) in /dev/shm or a nother disk it do not change anything.
There must be something in the VFS?
-Dieter
Signature · 4 lines
Dieter Nützel Graduate Student, Computer Science University of Hamburg @home: [email protected]
Quoted 16 lines · Dieter Nützel
Quoted 1 line · On Thursday, 20.12.201, 10:49 Helge Hafting wrote:
jlm wrote:
[-]
Quoted 7 lines
Quoted 3 lines
So I guess I don't really care what mode the hard drive is operating in (udma, mdma, dma or plain ide), I just don't want to have to go get a cup of coffee while the hard drive saves some data. Is there a "don't
Devices generally get the cpu before anything else. A good disk system don't need much cpu. Running IDE in PIO mode require a lot
of cpu though. Using any of the DMA modes avoids that.Amen..
Sorry, Helge sure you are right in theory but try dbench 32 (maybe bonnie/bonnie++) and playing an MP3/Ogg-Vorbis in parallel... That's my first test on any "new" kernel version.
Quoted 4 lines
Even with an 1 GHz Athlon II, 640 MB, U160 DDYS 18 GB, 10k IBM disk (on an AHA-2940UW) it stutters like mad. I am running all my kernel _with_ Robert Love's preempt + lock-break patches and it doesn't solve the problem. CPU load is (very) low but it do not work like it should.
try it with vanilla 2.4.17-rc1. I just did and i'm getting no
stuttering at all. nice -n 20 dbench 32. Worked quite nicely. Of
course i'm using an ext3 fs which is more important than your cpu speed
or ram. This kind of discussion has been talked about and argued over
many times in the past here already. Too many factors go into this and in the end, dbench is _Meant_ to preempt everything else. if you want a real test find a real program you really use and use it.
Quoted 32 lines
Quoted 4 lines
Quoted 2 lines
pre-empt the rest of the system" switch for the eide drives? Is there something fundamental/unique going on here that I'm missing?
dma, udma, etc. is that switch. It lets the cpu do other work (such as redrawing X) while the disk is busy. Plain ide is what you don't want.
See above the whole system show some bad hiccup.
Quoted 6 lines
The problem of waiting for other files or swapping while a really big write is going on is different. Get more drives, so the big writes go to one drive while you get stuff swapped in (or other file access) on other drive(s). The kernel is capable of getting fast response from one drive while another is completely bogged down with enormous writes.
Tried this already. Neither I put my test files (MP3/Ogg-Vorbis) in
/dev/shmor a nother disk it do not change anything.There must be something in the VFS?
-Dieter
Signature · 9 lines
Dieter Nützel Graduate Student, Computer Science University of Hamburg @home: [email protected] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Dieter N?tzel
Amen..
Sorry, Helge sure you are right in theory but try dbench 32 (maybe bonnie/bonnie++) and playing an MP3/Ogg-Vorbis in parallel... That's my first test on any "new" kernel version.Even with an 1 GHz Athlon II, 640 MB, U160 DDYS 18 GB, 10k IBM disk (on an AHA-2940UW) it stutters like mad. I am running all my kernel _with_ Robert Love's preempt + lock-break patches and it doesn't solve the problem. CPU load is (very) low but it do not work like it should.
I tried this on my 600MHz Athlon with 768MB of RAM and U160 DDYS 36GB 10Krpm IBM disk on a Adaptect 39160 I managed to get it not to stutter at all. I was also using preempt + lockbreak and a few others. The crucial patch appeared to be from Andrew Morton and it involved tuning the elevator to avoid read starvation. A significantly helpful hardware suggestion regarding the sound card and drivers came from Linus himself, though.
Linus and others pointed out that applications are able to cause some drivers to generate a large number of interrupts by using small buffers and unfriendly ioctl's, especially esd. My workaround was to change out sound hardware and disable esd. If this is happening to you, /proc/profile should show handle_IRQ_event() and schedule() very high up. On the other hand, this shows up as a steady drain on system resources and excessive system time, not stuttering or skipping.
Andrew, I don't have the URL for that still floating around. Can you point Dieter to it?
Cheers,
Bill
Quoted 3 lines · William Lee Irwin III
Andrew, I don't have the URL for that still floating around. Can you point Dieter to it?
It's here.
You need to run
elvtune -b N /dev/hdXX
where N=0 is "disable", N=1 is minimum read latency, N=6 is a reasonable setting.
drivers/block/elevator.c +56 -4
@@ -74,11 +74,10 @@ inline int bh_rq_in_between(struct buffereturn 0 ; } -int elevator_linus_merge (request_queue_t *q,struct request **req,struct list_head * head,struct buffer_head *bh,int rw, -int max_sectors) +int max_sectors,int max_bomb_segments) {struct list_head *entry = &q->queue_head;unsigned int count = bh->b_size >>9 , ret =ELEVATOR_NO_MERGE ;
@@ -116,6 +115,56 @@ int elevator_linus_merge(request_queue_t } } +/* +* If we failed to merge a read anywhere in the request +* queue, we really don't want to place it at the end +* of the list, behind lots of writes. So place it near +* the front. +* +* We don't want to place it in front of _all_ writes: that +* would create lots of seeking, and isn't tunable. +* We try to avoid promoting this read in front of existing +* reads. +* +* max_bomb_sectors becomes the maximum number of write +* requests which we allow to remain in place in front of +* a newly introduced read. We weight things a little bit, +* so large writes are more expensive than small ones, but it's +* requests which count, not sectors. +*/ +if (max_bomb_segments && rw ==READ && ret ==ELEVATOR_NO_MERGE ) { +int cur_latency =0 ; +struct request *const cur_request = *req; + + entry = head->next; +while (entry != &q->queue_head) { +struct request *__rq; + +if (entry == &q->queue_head) +BUG (); +if (entry == q->queue_head.next && + q->head_active && !q->plugged) +BUG (); + __rq =blkdev_entry_to_request (entry); + +if (__rq == cur_request) { +/* +* This is where the old algorithm placed it. +* There's no point pushing it further back, +* so leave it here, in sorted order. +*/ +break ; + } +if (__rq->cmd ==WRITE ) { + cur_latency +=1 + __rq->nr_sectors /64 ; +if (cur_latency >= max_bomb_segments) { + *req = __rq; +break ; + } + } + entry = entry->next; + } + }return ret; }
@@ -144,7 +193,7 @@ void elevator_linus_merge_req(struct reqint elevator_noop_merge (request_queue_t *q,struct request **req,struct list_head * head,struct buffer_head *bh,int rw, -int max_sectors) +int max_sectors,int max_bomb_segments) {struct list_head *entry;unsigned int count = bh->b_size >>9 ;
@@ -188,7 +237,7 @@ int blkelvget_ioctl(elevator_t * elevato output.queue_ID = elevator->queue_ID; output.read_latency = elevator->read_latency; output.write_latency = elevator->write_latency; - output.max_bomb_segments =0 ; + output.max_bomb_segments = elevator->max_bomb_segments;if (copy_to_user (arg, &output,sizeof (blkelv_ioctl_arg_t)))return -EFAULT ;
@@ -207,9 +256,12 @@ int blkelvset_ioctl(elevator_t * elevatoreturn -EINVAL ;if (input.write_latency <0 )return -EINVAL ; +if (input.max_bomb_segments <0 ) +return -EINVAL ; elevator->read_latency = input.read_latency; elevator->write_latency = input.write_latency; + elevator->max_bomb_segments = input.max_bomb_segments;return 0 ; }
drivers/block/ll_rw_blk.c +2 -1
@@ -690,7 +690,8 @@ again: }else if (q->head_active && !q->plugged) head = head->next; - el_ret = elevator->elevator_merge_fn (q, &req, head, bh, rw,max_sectors); + el_ret = elevator->elevator_merge_fn (q, &req, head, bh, + rw, max_sectors, elevator->max_bomb_segments);switch (el_ret) {case ELEVATOR_BACK_MERGE :
include/linux/elevator.h +13 -27
@@ -5,8 +5,9 @@ typedef void (elevator_fn) (struct reque struct list_head *, struct list_head *, int); -typedef int (elevator_merge_fn ) (request_queue_t *,struct request **,struct list_head *, -struct buffer_head *,int ,int ); +typedef int (elevator_merge_fn )(request_queue_t *,struct request **, +struct list_head *,struct buffer_head *bh, +int rw,int max_sectors,int max_bomb_segments);typedef void (elevator_merge_cleanup_fn ) (request_queue_t *,struct request *,int );
@@ -16,6 +17,7 @@ struct elevator_s {int read_latency;int write_latency; +int max_bomb_segments;elevator_merge_fn *elevator_merge_fn;elevator_merge_cleanup_fn *elevator_merge_cleanup_fn;
@@ -24,13 +26,13 @@ struct elevator_sunsigned int queue_ID; }; -int elevator_noop_merge (request_queue_t *,struct request **,struct list_head *,struct buffer_head *,int ,int ); -void elevator_noop_merge_cleanup (request_queue_t *,struct request *,int ); -void elevator_noop_merge_req (struct request *,struct request *); - -int elevator_linus_merge (request_queue_t *,struct request **,struct list_head *,struct buffer_head *,int ,int ); -void elevator_linus_merge_cleanup (request_queue_t *,struct request *,int ); -void elevator_linus_merge_req (struct request *,struct request *); +elevator_merge_fn elevator_noop_merge; +elevator_merge_cleanup_fn elevator_noop_merge_cleanup; +elevator_merge_req_fn elevator_noop_merge_req; + +elevator_merge_fn elevator_linus_merge; +elevator_merge_cleanup_fn elevator_linus_merge_cleanup; +elevator_merge_req_fn elevator_linus_merge_req; typedef struct blkelv_ioctl_arg_s { int queue_ID;
@@ -54,22 +56,6 @@ extern void elevator_init(elevator_t *,#define ELEVATOR_FRONT_MERGE 1#define ELEVATOR_BACK_MERGE 2 -/* -* This is used in the elevator algorithm. We don't prioritise reads -* over writes any more --- although reads are more time-critical than -* writes, by treating them equally we increase filesystem throughput. -* This turns out to give better overall performance. -- sct -*/ -#define IN_ORDER (s1,s2) \ - ((((s1)->rq_dev == (s2)->rq_dev && \ - (s1)->sector < (s2)->sector)) || \ - (s1)->rq_dev < (s2)->rq_dev) - -#define BHRQ_IN_ORDER (bh, rq) \ - ((((bh)->b_rdev == (rq)->rq_dev && \ - (bh)->b_rsector < (rq)->sector)) || \ - (bh)->b_rdev < (rq)->rq_dev) -static inline int elevator_request_latency (elevator_t * elevator,int rw) {int latency;
@@ -85,7 +71,7 @@ static inline int elevator_request_laten ((elevator_t) { \ 0, /* read_latency */ \ 0, /* write_latency */ \ - \ + 0, /* max_bomb_segments */ \ elevator_noop_merge, /* elevator_merge_fn */ \ elevator_noop_merge_cleanup, /* elevator_merge_cleanup_fn */ \ elevator_noop_merge_req, /* elevator_merge_req_fn */ \
@@ -95,7 +81,7 @@ static inline int elevator_request_laten ((elevator_t) { \ 8192, /* read passovers */ \ 16384, /* write passovers */ \ - \ + 0, /* max_bomb_segments */ \ elevator_linus_merge, /* elevator_merge_fn */ \ elevator_linus_merge_cleanup, /* elevator_merge_cleanup_fn */ \ elevator_linus_merge_req, /* elevator_merge_req_fn */ \
Quoted 4 lines · Andrew Morton
You need to run
elvtune -b N/dev/hdXX
where N=0 is "disable", N=1 is minimum read latency, N=6 is a reasonable setting.
I'm curious, why was max_bomb_segments dropped the last time it was in the tree ? I recall it happening, but the reason escapes me.
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs
Quoted 10 lines · Dave Jones
Quoted 4 lines · Andrew Morton
You need to run
elvtune -b N/dev/hdXX
where N=0 is "disable", N=1 is minimum read latency, N=6 is a reasonable setting.I'm curious, why was
max_bomb_segmentsdropped the last time it was in the tree ? I recall it happening, but the reason escapes me.
Fooled me too the first time, read Andrew's patch though. It isn't related at all.
Signature · 1 line
Jens Axboe
drivers/block/ll_rw_blk.c again: · line 694
Quoted 9 lines · Andrew Morton
drivers/block/ll_rw_blk.c +2
-1@@ -690,7 +690,8 @@ again: }else if (q->head_active && !q->plugged) head = head->next; - el_ret = elevator->elevator_merge_fn (q, &req, head, bh, rw,max_sectors); + el_ret = elevator->elevator_merge_fn (q, &req, head, bh, + rw, max_sectors, elevator->max_bomb_segments);
merge function can just grab max_bomb_segments ala
int mbs = q->elevator.max_bomb_segments
so no need to modify the merge functions.
Signature · 1 line
Jens Axboe