CFQ or BFQ scheduler and XFS
Hi list,
I have a question about CFQ scheduler and the old warning one can find on the faq page: "As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS".
Can I ask for more information about the bad interaction between CFQ and XFS, and especially why it does defeat filesystem parallelization? Is this warning still valid? What about the newer BFQ?
Note: I always used deadline or noop with XFS, but I am facing a disk with random read starvation when NCQ is enabled and a mixed sequential & random load happens. So far I saw that the only scheduler (somewhat) immune to the issue is CFQ, probably because it does not mix IO from multiple processes (it issue IO from one process at time, if I understand it correctly).
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 8 lines · Gionatan Danti
Hi list,
I have a question about CFQ scheduler and the old warning one can find on the faq page: "As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS".Can I ask for more information about the bad interaction between CFQ and XFS, and especially why it does defeat filesystem parallelization? Is this warning still valid? What about the newer BFQ?
CFQ doesn't understand that IO from different threads/tasks are related and so it cannot account for/control multi-threaded IO workloads. Given that XFS's journal IO is inherently a multi-threaded IO workload, CFQ IO accounting and throttling simply does not work properly with XFS or any other filesystem that decouples IO from the user task context that requires the IO to be done on it's behalf.
Quoted 5 lines
Note: I always used deadline or noop with XFS, but I am facing a disk with random read starvation when NCQ is enabled and a mixed sequential & random load happens. So far I saw that the only scheduler (somewhat) immune to the issue is CFQ, probably because it does not mix IO from multiple processes (it issue IO from one process at time, if I understand it correctly).
And, as per above, that's exactly why it doesn't work well with XFS...
Read starvation during sequential writes sounds more like a problem with the block layer writeback throttle or something you need to use an IO controller (blk-cgroups) to address, not try to fix with a IO scheduler...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Il 2022-01-23 23:52 Dave Chinner ha scritto:
Quoted 7 lines
CFQ doesn't understand that IO from different threads/tasks are related and so it cannot account for/control multi-threaded IO workloads. Given that XFS's journal IO is inherently a multi-threaded IO workload, CFQ IO accounting and throttling simply does not work properly with XFS or any other filesystem that decouples IO from the user task context that requires the IO to be done on it's behalf.
Hi Dave,
ah, so it forces all threads under the same time slice? I thought it was thread aware... Is it the same even for BFQ?
Quoted 6 lines
And, as per above, that's exactly why it doesn't work well with XFS...
Read starvation during sequential writes sounds more like a problem with the block layer writeback throttle or something you need to use an IO controller (blk-cgroups) to address, not try to fix with a IO scheduler...
To tell the truth, I simply disabled NCQ on the affected drive. I don't think anything is wrong with the block layer simply because I can constantly replicate the random read starvation by running, at the same time, the following fio commands against the raw block device:
fio --name=test --filename=/dev/sda --direct=1 --rw=read --runtime=30
#seq read
fio --name=test --filename=/dev/sda --direct=1 --rw=randread --runtime=30 #rnd read
The first fio sucks almost any resources, leaving the randread process with ~2 IOPS with frequent drops to 0. Using CFQ seems to avoid total starvation, but I then remembered it has some scalability and performance issues. Disabling NCQ solves the issue, at the cost of not using any hardware queue (obviously).
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 9 lines · Gionatan Danti
Il 2022-01-23 23:52 Dave Chinner ha scritto:
Quoted 7 lines
CFQ doesn't understand that IO from different threads/tasks are related and so it cannot account for/control multi-threaded IO workloads. Given that XFS's journal IO is inherently a multi-threaded IO workload, CFQ IO accounting and throttling simply does not work properly with XFS or any other filesystem that decouples IO from the user task context that requires the IO to be done on it's behalf.
Quoted 3 lines
Hi Dave,
ah, so it forces all threads under the same time slice? I thought it was thread aware...
No, the opposite. e.g. thread A does IO and then runs fsync(), which needs to force the log, that triggers CIL push work which runs in a different thread B. Thread A blocks waiting for the CIL push, thread B runs immediately. Thread B then issues log IO, but the IO from thread B is queued by CFQ because thread A's timeslice hasn't expired. Hence log force is delayed until thread A's timeslice expires, even though it is being done on behalf of thread A and thread A is blocked until the IO from Thread B is scheduled and completed.
Quoted 1 line
Is it the same even for BFQ?
No idea - I use noop for everything these days because IO schedulers often cause more problems than they solve on SSDs, sparse virtual machine images, thinly provisioned storage, etc....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Il 2022-01-24 22:13 Dave Chinner ha scritto:
Quoted 9 lines
No, the opposite. e.g. thread A does IO and then runs
fsync(), which needs to force the log, that triggers CIL push work which runs in a different thread B. Thread A blocks waiting for the CIL push, thread B runs immediately. Thread B then issues log IO, but the IO from thread B is queued by CFQ because thread A's timeslice hasn't expired. Hence log force is delayed until thread A's timeslice expires, even though it is being done on behalf of thread A and thread A is blocked until the IO from Thread B is scheduled and completed.
Hi Dave,
ok, I can see now why it is so bad (especially when multiple threads are involved).
Quoted 8 lines
No idea - I use noop for everything these days because IO schedulers often cause more problems than they solve on SSDs, sparse virtual machine images, thinly provisioned storage, etc....
Cheers,
Dave.
Right...
Thank you so much for the clear explanation.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8