Skip to contentSkip to conversation

Real-time problem due to IO congestion.

DISCUSSION Takashi Ikebe 5 participants 9 messages Jun 10, 2005 – Jun 13, 2005

lore
Share & tools
lore
#0 lore raw

Hello,
I have been encountering big real-time problem due to IO congestion, and I want some advices.

-The problem description-
There are 2 type processes in test environment.
1. The real-time needed process (run on with high static priority) The process wake up every 10ms, and wake up, write some log (the
test case is current CPU clock via tsc) to the file.

2. The process which make IO load
The process have large memory size, and kill the process with dumping. The process's memory area exceeds 70% of whole physical
RAM.(Actually 1.5GB memory area while whole RAM is 2GB)

Whenever during dumping, the real-time needed process sometimes stop for long time during write system call. (sometimes exceeds 1000ms) I tested every IO scheduler but the same problem occurs.
I also seek this problem into the code, and find that the stops are mainly occurring on blk_congestion_wait/get_request/get_request_wait functions located on drivers/block/ll_rw_blk.c.

-My assumption-
The design of IO(read/write) queue and queuing is not well match to real-time needed processes.
If there are many IO requests by low priority processes already, then the IO request by high priority process should wait until queue goes clean, and this cause some kind of priority inversions.

-My suggestion-
Add the new IO scheduler or change current IO scheduler to reflect process's priority on queuing.

I don't know my assumption and suggestion are correct and you like, would you give me some advices?

Thank you.

Signature · 1 line
Takashi Ikebe
#1 lore raw
Quoted 29 lines · Takashi Ikebe

Hello,
I have been encountering big real-time problem due to IO congestion, and I want some advices.

-The problem description-
There are 2 type processes in test environment.
1. The real-time needed process (run on with high static priority) The process wake up every 10ms, and wake up, write some log (the
test case is current CPU clock via tsc) to the file.

2. The process which make IO load
The process have large memory size, and kill the process with dumping. The process's memory area exceeds 70% of whole physical
RAM.(Actually 1.5GB memory area while whole RAM is 2GB)

Whenever during dumping, the real-time needed process sometimes stop for long time during write system call. (sometimes exceeds 1000ms) I tested every IO scheduler but the same problem occurs.
I also seek this problem into the code, and find that the stops are mainly occurring on blk_congestion_wait/get_request/get_request_wait functions located on drivers/block/ll_rw_blk.c.

-My assumption-
The design of IO(read/write) queue and queuing is not well match to real-time needed processes.
If there are many IO requests by low priority processes already, then the IO request by high priority process should wait until queue goes clean, and this cause some kind of priority inversions.

Quoted 6 lines

-My suggestion-
Add the new IO scheduler or change current IO scheduler to reflect process's priority on queuing.

I don't know my assumption and suggestion are correct and you like, would you give me some advices?

This basically needs io priorities to work, so that request allocation is prioritized as well. I didn't actually add request allocation groups in the cfq-ts posted with priority support, however I have some patches from years ago that did so. I'll see if I can find the time to brush those off.

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 8 lines · Jens Axboe

On Fri, Jun 10 2005, Takashi Ikebe wrote:

This basically needs io priorities to work, so that request allocation is prioritized as well. I didn't actually add request allocation groups in the cfq-ts posted with priority support, however I have some patches from years ago that did so. I'll see if I can find the time to brush those off.

As you and andrew said, basically application based approach seems reasonable, but I'm so interesting your patch, if you have time, please show me :-)

Thank you.

Signature · 1 line
Takashi Ikebe
#3 lore raw
Quoted 10 lines · Takashi Ikebe
Quoted 8 lines · Jens Axboe

On Fri, Jun 10 2005, Takashi Ikebe wrote:

This basically needs io priorities to work, so that request allocation is prioritized as well. I didn't actually add request allocation groups in the cfq-ts posted with priority support, however I have some patches from years ago that did so. I'll see if I can find the time to brush those off.

Quoted 3 lines

As you and andrew said, basically application based approach seems reasonable,
but I'm so interesting your patch, if you have time, please show me :-)

Take a look at the lock-free ringbuffers in JACK. No point reinventing the wheel...

I think it's a bit strange that the disk IO issue comes up so often in these RT discussions, it's something of a red herring, because there's rarely an RT constraint in getting the data to disk; the RT constraint is in getting the data from the device to memory.

Lee

#4 lore raw
In reply to Takashi Ikebe
Quoted 6 lines · Takashi Ikebe

There are 2 type processes in test environment.
1. The real-time needed process (run on with high static priority) The process wake up every 10ms, and wake up, write some log (the
test case is current CPU clock via tsc) to the file.

Quoted 7 lines

2. The process which make IO load
The process have large memory size, and kill the process with dumping. The process's memory area exceeds 70% of whole physical
RAM.(Actually 1.5GB memory area while whole RAM is 2GB)

Whenever during dumping, the real-time needed process sometimes stop for long time during write system call. (sometimes exceeds 1000ms)

The writeback code does attempt to give some preference to realtime tasks (in get_dirty_limits()), but it can only work up to a point.

Frankly, your application is poorly designed. If you want sub-10ms responsiveness you shouldn't be doing disk I/O. The realtime task should hand the data off to a non-realtime task for writeout, with suitable amounts of buffering in between.

#5 lore raw

I see.
The program which I tested is just sample, and I wanted to know the phenomena is spec or bug. I also understand that this problem is spec, and need to apply some buffering to such applications.

Thank you.

Quoted 23 lines · Andrew Morton
Quoted 5 lines · Takashi Ikebe

There are 2 type processes in test environment.
1. The real-time needed process (run on with high static priority) The process wake up every 10ms, and wake up, write some log (the
test case is current CPU clock via tsc) to the file.

Quoted 7 lines

2. The process which make IO load
The process have large memory size, and kill the process with dumping. The process's memory area exceeds 70% of whole physical
RAM.(Actually 1.5GB memory area while whole RAM is 2GB)

Whenever during dumping, the real-time needed process sometimes stop for long time during write system call. (sometimes exceeds 1000ms)

The writeback code does attempt to give some preference to realtime tasks (in get_dirty_limits()), but it can only work up to a point.

Frankly, your application is poorly designed. If you want sub-10ms responsiveness you shouldn't be doing disk I/O. The realtime task should hand the data off to a non-realtime task for writeout, with suitable amounts of buffering in between.

Signature · 1 line
Takashi Ikebe
#6 lore raw

(please don't top post)

Quoted 6 lines · Takashi Ikebe

I see.
The program which I tested is just sample, and I wanted to know the phenomena is spec or bug.
I also understand that this problem is spec, and need to apply some buffering to such applications.

Additionally, following up on Andrew, even with prioritized request allocations you could get equally long stuck if you just had lots of high prio allocaters queueing io. So rethinking the setup is definitely good advice.

Signature · 1 line
Jens Axboe
#7 lore raw
In reply to Takashi Ikebe
Quoted 7 lines · Takashi Ikebe

I see.
The program which I tested is just sample, and I wanted to know the phenomena is spec or bug.
I also understand that this problem is spec, and need to apply some buffering to such applications.

There is an alternative. Get a separate disk just for the RT-job. You can then run your very heavy IO on other disks, the RT disk won't be delayed by that.

Helge Hafting

#8 lore raw

Thank you for your kind advices!

Our system has characteristics such as embedded system, and, unfortunately, such systems have the restriction that available disk is only 1(raid 1 mirrored one). But the system has large memory (maximum 16GB), and SMP. (such as blade type system)

I think this problem can also be solved by modifying page cache transactions(assume there is huge page cache!), and this may guarantee somewhat real-time write even on single normal disk. but this may be only the answer in case of huge page-cache, and may be not for user desktops.

what I'm considering is that the this approach is acceptable by kernel community or not?? (as you mentioned, kernel modify is one approach, and app level buffering is another approach). If I or somebody made some patch, is there any possibility to be accepted? or this type of problem should not go into kernel?

Quoted 17 lines · Helge Hafting
Quoted 4 lines · Takashi Ikebe

I see.
The program which I tested is just sample, and I wanted to know the phenomena is spec or bug. I also understand that this problem is spec, and need to apply some buffering to such applications.

There is an alternative. Get a separate disk just for the RT-job. You can then run your very heavy IO on other disks, the RT disk won't be delayed by that.

Helge Hafting
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 6 lines
Takashi Ikebe
NTT Network Service Systems Laboratories
9-11, Midori-Cho 3-Chome Musashino-Shi,
Tokyo 180-8585 Japan
Tel : +81 422 59 4246, Fax : +81 422 60 4012
e-mail : [email protected]