io priorities?

Hi folks, i've got IO problems...

I've got a computer with one IDE disk. There are a couple of processes writing big telemetry logs, totally not time-critical. Then there's one process periodically writing a tiny little state file, about 150 bytes. The process with the state file wants to sync its file to disk and then quickly be back and running. When it calls fsync() it has to wait for other processes' less-important pending I/O before it gets to complete, and it's not unusual for this to take 30 seconds (CF over IDE), which is too slow for my application.

This happens with both 2.4.23 and 2.6.6 (as, cfq, & deadline). I havent tried to tune the schedulers at all, just using the default values.

I've tried five methods of syncing the file: sync(), fsync(), fdatasync(), and opening with O_SYNC or O_DSYNC. All take about the same amount of time.

I've tried it on my target machine (Compact Flash via pio IDE) and on my development machine (regular hard drive via udma5 IDE). Same results qualitatively speaking.

I read Jens Axboe's thread about cfq + io priorities, and it sounds perfect! I could give my one time-critical process high io priority and it should preempt the others and life would be fine. But all the l-k traffic i've found about this is from back in November. Did this work go anywhere?

Any suggestions on fixes or workarounds?

I'm stumped.

--
Sebastian

Periodically fsync() the logs so there isn't soooo much stuff to write. In fact, a simple sync() call about once every few seconds should make everything work, i.e. ...

main()

{
    for(;;)
    {
        sync();
        sleep(3);
    }
}

... in another task should fix it. We no longer have update running like the old unixes. Now there is bdflush that cares only about kernel buffers getting in short supply. If you want your files to be sync(ed) quickly, you need to keep the in-kernel buffers short which means you need to do it yourself.

Before everybody chimes in.... just try it. You'll probably like it.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.

Yup. Alternatively, set /proc/sys/vm/dirty_ratio and /proc/sys/vm/dirty_background_ratio to really small values. Say, 2 and 1.

Andrew Morton <[email protected]> wrote:
] "Richard B. Johnson" <[email protected]> wrote:
] > Periodically fsync() the logs so there isn't soooo much stuff to
] > write. In fact, a simple sync() call about once every few seconds
] > should make everything work,
]
] Yup. Alternatively, set /proc/sys/vm/dirty_ratio and /proc/sys/vm/dirty_background_ratio ] to really small values. Say, 2 and 1.

Thanks guys, that sort of fixes it, but not really. Occasionally i get a big load of IO (copying tens of megs of log files around), and even if i synced a second before, now there's tens of seconds of IO in the queue, and my next fsync() still drags on.

Really i just want this one process to go to the front of the queue, any time it has IO to do, independent of what the rest of the system is doing.

--
Sebastian