rt file i/o
hi all,
having read the howto build an rt-application, i am trying to think of a way to handle disk i/o in my application.
the howto suggest to avoid file i/o in non-rt helper threads, but to move them to another process. while the howto suggests using network socket, i'm curious, if other ipc primitives like shared memory would give better throughput? or would i experience any issues, like is it possible to lock shared memory to physical ram?
thanks, tim
Signature · 5 lines
[email protected] http://tim.klingt.org Art is either a complaint or do something else John Cage quoting Jasper Johns
Quoted 11 lines · Tim Blechmann
hi all,
having read the howto build an rt-application, i am trying to think of a way to handle disk i/o in my application.
the howto suggest to avoid file i/o in non-rt helper threads, but to move them to another process. while the howto suggests using network socket, i'm curious, if other ipc primitives like shared memory would give better throughput? or would i experience any issues, like is it possible to lock shared memory to physical ram?
Quoted 9 lines
thanks, tim
--
[email protected]
http://tim.klingt.orgArt is either a complaint or do something else
John Cage quoting Jasper Johns
Tim,
I'm not a developer and cannot really address your specific
question but for my use of the rt-kernel I put important rt-audio files on a 1394 drive and give the 1394 driver higher priorities using the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394 drives recording and playing back 48 channels in Ardour.
While I agree you should do everything the right way technically in
the code maybe part of your solution is outside of the app you are writing and in the use of these support tools?
Take care,
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines
I'm not a developer and cannot really address your specific
question but for my use of the rt-kernel I put important rt-audio files on a 1394 drive and give the 1394 driver higher priorities using the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394 drives recording and playing back 48 channels in Ardour.While I agree you should do everything the right way technically in
the code maybe part of your solution is outside of the app you are writing and in the use of these support tools?
well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
according to this definition, ardour is not an rt-safe application ... as for rt-safety, i prefer analyzing the latency hotspots instead of relying on test results
tim
Signature · 6 lines
[email protected] http://tim.klingt.org Avoid the world, it's just a lot of dust and drag and means nothing in the end. Jack Kerouac
Quoted 6 lines · Tim Blechmann
Quoted 6 lines
I'm not a developer and cannot really address your specific
question but for my use of the rt-kernel I put important rt-audio files on a 1394 drive and give the 1394 driver higher priorities using the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394 drives recording and playing back 48 channels in Ardour.
Quoted 7 lines
Quoted 3 lines
While I agree you should do everything the right way technically in
the code maybe part of your solution is outside of the app you are writing and in the use of these support tools?well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
I would be supprised if the rt howto states that page-faults in non-rt threads is a critical problem - that would not significantly impact RT performance - atelast not the worst case - it will (as every other system load) impact the average case. so having a non-rt thread reading disk-files to a buffer and a rt-thread processing this buffer should be perfectly fine.
hofrat
Quoted 9 lines
Quoted 3 lines
well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
I would be supprised if the rt howto states that page-faults in non-rt threads is a critical problem - that would not significantly impact RT performance - atelast not the worst case - it will (as every other system load) impact the average case. so having a non-rt thread reading disk-files to a buffer and a rt-thread processing this buffer should be perfectly fine.
that is what i thought for years ... but according to [1] a page fault in an rt process freezes the entire process with both rt and non-rt threads until the page fault is handled ...
tim
[1]
http://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application#Latencies_caused_by_Page-faults
Signature · 6 lines
[email protected] http://tim.klingt.org Your mind will answer most questions if you learn to relax and wait for the answer. William S. Burroughs
Quoted 10 lines · Tim Blechmann
Quoted 9 lines
Quoted 3 lines
well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
I would be supprised if the rt howto states that page-faults in non-rt threads is a critical problem - that would not significantly impact RT performance - atelast not the worst case - it will (as every other system load) impact the average case. so having a non-rt thread reading disk-files to a buffer and a rt-thread processing this buffer should be perfectly fine.
Quoted 4 lines
that is what i thought for years ... but according to [1] a page fault in an rt process freezes the entire process with both rt and non-rt threads until the page fault is handled ...
I proved to my self only a few days ago that I know nothing about RT_Preempt so at the risk that this is completly wrong - for a simple test I added a reader/writer thread in cyclictest.
with SCHED_OTHER or priority below 50 with SCHED_FIFO the impact is as expected with priority above 50 (left all the irq-threads at there usual default) I see no impact at all in the worst case (average - as can be expected goes up a bit compared to an idle system).
here are the results of a short run - the effect pops up imediately for SCHED_OTHER or -p 48, the -p 99 run is about 15 minutes.
/* SCHED_OTHER */
rtl26:~/rt/rt-tests# ./cyclictest -n 0.71 0.27 0.16 1/128 6907
T: 0 ( 6906) P: 0 I:1000 C: 9383 Min: 2 Act: 52 Avg: 315 Max: 73704
/* priority below any I/O activity in the kernel */
rtl26:~/rt/rt-tests# ./cyclictest -n -p 48 0.44 0.25 0.16 1/128 6911
T: 0 ( 6910) P:48 I:1000 C: 8610 Min: 1 Act: 2 Avg: 322 Max: 73607
/* priority above any I/O activity in the kernel */
rtl26:~/rt/rt-tests.mod# ./cyclictest -n -p 99 1.60 1.59 1.23 1/128 7314
T: 0 ( 7079) P:99 I:1000 C:1067096 Min: 1 Act: 2 Avg: 2 Max: 31
The thread added simply opens a large file (linux-2.6.31.tar.bz2, 62MB), reading 1k at a time, and writing it to a file in /tmp in an while(1) loop - the thread runs with default attr thus PTHREAD_SCOPE_SYSTEM as SCHED_OTHER thread. so maybe someone with more insight could comment on this "good" result.
HW: UP AMD Sempron 1.6GHz, NN IDE disk, 256MB RAM
SW: Debian 5.0.3 (Default Desktop) kernel 2.6.31.4-rt14
thx!
hofrat
Quoted 10 lines · On Wed, 04 Nov 2009 18:32:41 +0100 Tim Blechmann <[email protected]> wrote:
Quoted 9 lines
Quoted 3 lines
well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
I would be supprised if the rt howto states that page-faults in non-rt threads is a critical problem - that would not significantly impact RT performance - atelast not the worst case - it will (as every other system load) impact the average case. so having a non-rt thread reading disk-files to a buffer and a rt-thread processing this buffer should be perfectly fine.
Quoted 4 lines
that is what i thought for years ... but according to [1] a page fault in an rt process freezes the entire process with both rt and non-rt threads until the page fault is handled ...
No, we need to get that fixed in the wiki. It may have been the case that the entire process was frozen while a fault was being handled, but I don't believe that is the case now. If you're in a thread on core0 and a sibling thread on core1 faults, you shouldn't be impacted unless you're trying to reference the same memory.
Clark
Clark Williams <[email protected]> wrote on 11/04/2009 03:44:13 PM:
Quoted 7 lines
No, we need to get that fixed in the wiki. It may have been the case that the entire process was frozen while a fault was being handled, but I don't believe that is the case now. If you're in a thread on core0 and a sibling thread on core1 faults, you shouldn't be impacted unless you're trying to reference the same memory.
Clark
I had seen this myself in the wiki and have been telling many people about it. I finally tested this a couple of months ago and found out it was not true. I tried having a thread malloc and touch memory, write to the disk, and read from the disk. No matter what I did, I couldn't affect a second thread.
I have read briefly on some of the differences with the current and previous user model. I don't know all the facts, but I believe that it used to be a user library. At that time, I'm sure that a page fault would effect every thread.
Shane
Quoted 18 lines · Clark Williams
Quoted 10 lines · On Wed, 04 Nov 2009 18:32:41 +0100 Tim Blechmann <[email protected]> wrote:
Quoted 9 lines
Quoted 3 lines
well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
I would be supprised if the rt howto states that page-faults in non-rt threads is a critical problem - that would not significantly impact RT performance - atelast not the worst case - it will (as every other system load) impact the average case. so having a non-rt thread reading disk-files to a buffer and a rt-thread processing this buffer should be perfectly fine.
Quoted 4 lines
that is what i thought for years ... but according to [1] a page fault in an rt process freezes the entire process with both rt and non-rt threads until the page fault is handled ...
Quoted 5 lines
No, we need to get that fixed in the wiki. It may have been the case that the entire process was frozen while a fault was being handled, but I don't believe that is the case now. If you're in a thread on core0 and a sibling thread on core1 faults, you shouldn't be impacted unless you're trying to reference the same memory.
well, then my application seems to be real-time safe ... it would be nice if the wiki would reflect the current state of the rt-preempt kernel, though ...
thnx, tim
Signature · 6 lines
[email protected] http://tim.klingt.org All we composers really have to work with is time and sound - and sometimes I'm not even sure about sound. Morton Feldman
Hi,
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
that is what i thought for years ... but according to [1] a page fault in an rt process freezes the entire process with both rt and non-rt threads until the page fault is handled ...
No, we need to get that fixed in the wiki. It may have been the case that the entire process was frozen while a fault was being handled, but I don't believe that is the case now. If you're in a thread on core0 and a sibling thread on core1 faults, you shouldn't be impacted unless you're trying to reference the same memory.
Quoted 3 lines
well, then my application seems to be real-time safe ... it would be nice if the wiki would reflect the current state of the rt-preempt kernel, though ...
Well, I wrote this part of the wiki quite some time ago, and yes it might be outdated on some points.
I will look into it, and update it in a couple of days.
Kind regards,
Remy
Quoted 17 lines · Nicholas Mc Guire
Quoted 6 lines · Tim Blechmann
Quoted 6 lines
I'm not a developer and cannot really address your specific
question but for my use of the rt-kernel I put important rt-audio files on a 1394 drive and give the 1394 driver higher priorities using the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394 drives recording and playing back 48 channels in Ardour.Quoted 8 lines
Quoted 4 lines
While I agree you should do everything the right way technically in
the code maybe part of your solution is outside of the app you are writing and in the use of these support tools?well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
Quoted 7 lines
I would be supprised if the rt howto states that page-faults in non-rt threads is a critical problem - that would not significantly impact RT performance - atelast not the worst case - it will (as every other system load) impact the average case. so having a non-rt thread reading disk-files to a buffer and a rt-thread processing this buffer should be perfectly fine.
Ack. I agree with Herr Hofrat, and would only add, that it is helpful to pre-allocate and mlock() memory you use to pipe data from RT tasks to the I/O subsystem.
It is possible to trigger delays in a priority-agnostic manner when a large number of tasks end up all hammering heavily on malloc().
Sven
Quoted 6 lines
hofrat
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 19 lines · Tim Blechmann
Quoted 9 lines
I'm not a developer and cannot really address your specific question but for my use of the rt-kernel I put important rt-audio files on a 1394 drive and give the 1394 driver higher priorities using the IRQ scheduling tools. I don't have any trouble running 2 or 3 1394 drives recording and playing back 48 channels in Ardour.
While I agree you should do everything the right way technically in the code maybe part of your solution is outside of the app you are writing and in the use of these support tools?
well, if i understand the rt howto correctly, _no_ disc access is allowed, neither from rt nor from non-rt threads, since it may produce page faults, which introduce latencies ...
according to this definition, ardour is not an rt-safe application ... as for rt-safety, i prefer analyzing the latency hotspots instead of relying on test resultstim
Quoted 7 lines
--
[email protected]
http://tim.klingt.orgAvoid the world, it's just a lot of dust and drag and means nothing in the end.
Jack Kerouac
In the limit you may be right. Possibly unbounded disk IO would never be real-time safe. However in practice I see xruns when I use normal priorities and none when I give my 1394 driver higher priorities.
Logically it doesn't make sense to me that *no* disk access is allowed. I think the rt-threads just do their work and the drives catch up when they can. Apps like Ardour are reading and writing the hard drive all the time. Lots of files are open and streaming, in my case maybe 40 files running for hours.
I think that disk access in an rt-thread make no sense at it cannot response real-time, but non-rt disk activity just gets interrupted like any other non-rt process, right?
Good luck with what you are up to.
Cheers,
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · Tim Blechmann
hi all,
having read the howto build an rt-application, i am trying to think of a way to handle disk i/o in my application.
the howto suggest to avoid file i/o in non-rt helper threads, but to move them to another process. while the howto suggests using network socket, i'm curious, if other ipc primitives like shared memory would give better throughput? or would i experience any issues, like is it possible to lock shared memory to physical ram?
this is probably a bit off topic - but both RTLinux/GPL (RTLFS) and L4 have implemented RT IDE drivers - limited to dedicated discs - the performance is really bad (aprox. 800k/s) - so determinism on a device that really lives from cache/pre-fetch and large-block access is very expensive. What such a driver can give you at best is a guaranteed bandwidth, and a worst case latency that is not really too atractive (tens of milliseconds)
So probably the best option is to have a user-space/non-RT driver access the disk and pre-fetch/cache a "large" chunk of what you want to read/write to de-couple RT from non-RT - this has been the model used i.e. for RAT (Real-time Audio Tools see proceedings of the 6th Real Time Linux Workshop - Singapur) quite successfully - as long as the I/O rate your RT-app expects is sufficiently much smaller than the I/O capability of the device driver AND you ensure the bandwidth (i.e. by raising the respective irq-thread priority, and restricting other I/O to that controler/disk) this is actually a quite deterministic approach provided your buffering is not too small.
Generally you will not be able to analyze paths of complex hardware/software like disk-subsystems in a GPOS to the point were you can give absolute guarantees - what you can achieve though is statistic guarantees, and in combination with monitoring of the cached data you can (atleast for some access patters) give guarantees about the behavior of the system.
hofrat
RAT http://www.osadl.org/Papers.rtlws-2004-papers.0.html#PAPER_2590
RTLFS http://www.osadl.org/Papers.rtlws-2004-papers.0.html#PAPER_1889