fuse: max_background and congestion_threshold settings
Hello,
Could someone explain to me the meaning of the max_background and congestion_threshold settings of the fuse module?
At first I assumed that max_background specifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking at fs/fuse/dev.c, it looks as if not every request is included in this number.
I also figured out that if the number of background requests (whatever they are) exceeds the congestion threshold, fuse calls set_bdi_congested() for the backing device. But what does this do? And does this become a no-op if there is no backing device?
Thanks,
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Hi,
Quoted 9 lines · Nikolaus Rath
Hello,
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.
fuse uses max_background for cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's where max_background steps in.
Quoted 4 lines
I also figured out that if the number of background requests (whatever they are) exceeds the congestion threshold, fuse calls
set_bdi_congested()for the backing device. But what does this do?
AFAIU, this is a hint for reclaimer to avoid busy loop:
Quoted 10 lines
/*
* If kswapd scans pages marked marked for immediate
* reclaim and under writeback (nr_immediate), it implies
* that pages are cycling through the LRU faster than
* they are written so also forcibly stall.
*/
if (nr_immediate&¤t_may_throttle())
congestion_wait(BLK_RW_ASYNC, HZ/10);
Quoted 2 lines
And
does this become a no-op if there is no backing device?
current->backing_dev_info exists (and helps to control writeback) even if there is no "real" backing device.
Thanks,
Maxim
Hi Maxim,
Quoted 9 lines · Maxim Patlasov
Quoted 7 lines · Nikolaus Rath
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.
Quoted 9 lines
fuse uses
max_backgroundfor cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's wheremax_backgroundsteps in.
Ah, that makes sense. Are these two cases meant as examples, or is that an exhaustive list? Because I would have thought that other cases should be writing of cached data (when writeback caching is enabled), and asynchronous I/O from userspace...?
Also, I am not sure what you mean with async processing of direct I/O. Shouldn't direct I/O always go directly to the file-system? If so, how can it be processed asynchronously?
Best,
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 20 lines · Nikolaus Rath
Hi Maxim,
Quoted 8 lines · Maxim Patlasov
Quoted 7 lines · Nikolaus Rath
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.Quoted 9 lines
fuse uses
max_backgroundfor cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's wheremax_backgroundsteps in.
Quoted 4 lines
Ah, that makes sense. Are these two cases meant as examples, or is that an exhaustive list? Because I would have thought that other cases should be writing of cached data (when writeback caching is enabled), and asynchronous I/O from userspace...?
I think that's exhaustive list, but I can miss something.
As for writing of cached data, that definitely doesn't go through background requests. Here we rely on flusher: fuse will allocate as many requests as the flusher wants to writeback.
Buffered AIO READs actually block in submit_io until fully processed. So it's just another example of "natural" limitation I told above. Buffered AIO WRITEs go through writeback mechanics anyway, so here again we rely on flusher behaving reasonable. And finally, direct AIO does go through fuse background requests as I wrote: "1) async processing of direct IO;"
Quoted 4 lines
Also, I am not sure what you mean with async processing of direct I/O. Shouldn't direct I/O always go directly to the file-system? If so, how can it be processed asynchronously?
That's a nice optimization we implemented a few years ago: having incoming sync direct IO request of 1MB size, kernel fuse splits it into eight 128K requests and starts processing them in async manner, waiting for the completion of all of them before completing that incoming 1MB requests. This boosts performance tremendously if userspace fuse daemon is able to efficiently process many requests "in parallel". This optimization is implemented using background fuse requests. Otherwise, having 1GB incoming request, we would obediently allocate 8K fuse requests in one shot -- too dangerous and not good for latency.
Thanks,
Maxim
Quoted 29 lines · Maxim Patlasov
Quoted 20 lines · Nikolaus Rath
Hi Maxim,
Quoted 8 lines · Maxim Patlasov
Quoted 7 lines · Nikolaus Rath
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.Quoted 9 lines
fuse uses
max_backgroundfor cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's wheremax_backgroundsteps in.Quoted 4 lines
Ah, that makes sense. Are these two cases meant as examples, or is that an exhaustive list? Because I would have thought that other cases should be writing of cached data (when writeback caching is enabled), and asynchronous I/O from userspace...?
I think that's exhaustive list, but I can miss something.
Quoted 7 lines
As for writing of cached data, that definitely doesn't go through background requests. Here we rely on flusher: fuse will allocate as many requests as the flusher wants to writeback.
Buffered AIO READs actually block in
submit_iountil fully processed. So it's just another example of "natural" limitation I told above.
Not sure I understand. What is it that's blocking? It can't be the userspace process, because then it wouldn't be asynchronous I/O...
Quoted 9 lines
Quoted 3 lines
Also, I am not sure what you mean with async processing of direct I/O. Shouldn't direct I/O always go directly to the file-system? If so, how can it be processed asynchronously?
That's a nice optimization we implemented a few years ago: having incoming sync direct IO request of 1MB size, kernel fuse splits it into eight 128K requests and starts processing them in async manner, waiting for the completion of all of them before completing that incoming 1MB requests.
I see. But why isn't that also done for regular (non-direct) IO?
Thanks,
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 33 lines · Nikolaus Rath
Quoted 32 lines · Maxim Patlasov
Quoted 20 lines · Nikolaus Rath
Hi Maxim,
Quoted 8 lines · Maxim Patlasov
Quoted 7 lines · Nikolaus Rath
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.Quoted 9 lines
fuse uses
max_backgroundfor cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's wheremax_backgroundsteps in.Quoted 4 lines
Ah, that makes sense. Are these two cases meant as examples, or is that an exhaustive list? Because I would have thought that other cases should be writing of cached data (when writeback caching is enabled), and asynchronous I/O from userspace...?
I think that's exhaustive list, but I can miss something.
As for writing of cached data, that definitely doesn't go through background requests. Here we rely on flusher: fuse will allocate as many requests as the flusher wants to writeback.
Quoted 5 lines
Quoted 3 lines
Buffered AIO READs actually block in
submit_iountil fully processed. So it's just another example of "natural" limitation I told above.Not sure I understand. What is it that's blocking? It can't be the userspace process, because then it wouldn't be asynchronous I/O...
Surprise! Alas, Linux kernel does NOT process buffered AIO reads in async manner. You can verify it yourself by strace-ing a simple program looping over io_submit + io_getevents: for direct IO (as expected) io_submit returns immediately while io_getevents waits for actual IO; in contrast, for buffered IO (surprisingly) io_submit waits for actual IO while io_getevents returns immediately. Presumably, people are supposed to use mmap-ed read/writes rather than buffered AIO.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Also, I am not sure what you mean with async processing of direct I/O. Shouldn't direct I/O always go directly to the file-system? If so, how can it be processed asynchronously?
That's a nice optimization we implemented a few years ago: having incoming sync direct IO request of 1MB size, kernel fuse splits it into eight 128K requests and starts processing them in async manner, waiting for the completion of all of them before completing that incoming 1MB requests.
I see. But why isn't that also done for regular (non-direct) IO?
Regular READs are helped by async read-ahead. Regular writes go through writeback mechanics: flusher calls fuse_writepages() and the latter submits as many async write requests as needed. Everything looks fine. (but as I wrote those async requests are not under fuse max_backgroung control).
Thanks,
Maxim
Quoted 3 lines
Thanks,
-Nikolaus
Quoted 41 lines · Maxim Patlasov
Quoted 33 lines · Nikolaus Rath
Quoted 32 lines · Maxim Patlasov
Quoted 20 lines · Nikolaus Rath
Hi Maxim,
Quoted 8 lines · Maxim Patlasov
Quoted 7 lines · Nikolaus Rath
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.Quoted 9 lines
fuse uses
max_backgroundfor cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's wheremax_backgroundsteps in.Quoted 4 lines
Ah, that makes sense. Are these two cases meant as examples, or is that an exhaustive list? Because I would have thought that other cases should be writing of cached data (when writeback caching is enabled), and asynchronous I/O from userspace...?
I think that's exhaustive list, but I can miss something.
As for writing of cached data, that definitely doesn't go through background requests. Here we rely on flusher: fuse will allocate as many requests as the flusher wants to writeback.
Quoted 5 lines
Quoted 3 lines
Buffered AIO READs actually block in
submit_iountil fully processed. So it's just another example of "natural" limitation I told above.Not sure I understand. What is it that's blocking? It can't be the userspace process, because then it wouldn't be asynchronous I/O...
Quoted 8 lines
Surprise! Alas, Linux kernel does NOT process buffered AIO reads in async manner. You can verify it yourself by strace-ing a simple program looping over
io_submit+io_getevents: for direct IO (as expected)io_submitreturns immediately whileio_geteventswaits for actual IO; in contrast, for buffered IO (surprisingly)io_submitwaits for actual IO whileio_geteventsreturns immediately. Presumably, people are supposed to use mmap-ed read/writes rather than buffered AIO.
What about buffered, asynchronous writes when writeback cache is disabled? It sounds as if io_submit does not block (so userspace could create an unlimited number), nor can the kernel coalesce them (since writeback caching is disabled).
Thanks!
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 49 lines · Nikolaus Rath
Quoted 40 lines · Maxim Patlasov
Quoted 33 lines · Nikolaus Rath
Quoted 32 lines · Maxim Patlasov
Quoted 20 lines · Nikolaus Rath
Hi Maxim,
Quoted 8 lines · Maxim Patlasov
Quoted 7 lines · Nikolaus Rath
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.Quoted 9 lines
fuse uses
max_backgroundfor cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's wheremax_backgroundsteps in.Quoted 4 lines
Ah, that makes sense. Are these two cases meant as examples, or is that an exhaustive list? Because I would have thought that other cases should be writing of cached data (when writeback caching is enabled), and asynchronous I/O from userspace...?
I think that's exhaustive list, but I can miss something.
As for writing of cached data, that definitely doesn't go through background requests. Here we rely on flusher: fuse will allocate as many requests as the flusher wants to writeback.
Quoted 5 lines
Quoted 3 lines
Buffered AIO READs actually block in
submit_iountil fully processed. So it's just another example of "natural" limitation I told above.Not sure I understand. What is it that's blocking? It can't be the userspace process, because then it wouldn't be asynchronous I/O...
Quoted 8 lines
Surprise! Alas, Linux kernel does NOT process buffered AIO reads in async manner. You can verify it yourself by strace-ing a simple program looping over
io_submit+io_getevents: for direct IO (as expected)io_submitreturns immediately whileio_geteventswaits for actual IO; in contrast, for buffered IO (surprisingly)io_submitwaits for actual IO whileio_geteventsreturns immediately. Presumably, people are supposed to use mmap-ed read/writes rather than buffered AIO.
Quoted 4 lines
What about buffered, asynchronous writes when writeback cache is disabled? It sounds as if
io_submitdoes not block (so userspace could create an unlimited number), nor can the kernel coalesce them (since writeback caching is disabled).
I've never looked closely at it. Do you have a particular use case or concern?
Quoted 4 lines
Thanks!
-Nikolaus
Hi Maxim,
Quoted 39 lines · Maxim Patlasov
Quoted 39 lines
Quoted 39 lines
Quoted 26 lines
Quoted 26 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Could someone explain to me the meaning of the
max_backgroundandcongestion_thresholdsettings of the fuse module?At first I assumed that
max_backgroundspecifies the maximum number of pending requests (i.e., requests that have been send to userspace but for which no reply was received yet). But looking atfs/fuse/dev.c, it looks as if not every request is included in this number.Quoted 9 lines
fuse uses
max_backgroundfor cases where the total number of simultaneous requests of given type is not limited by some other natural means. AFAIU, these cases are: 1) async processing of direct IO; 2) read-ahead. As an example of "natural" limitation: when userspace process blocks on a sync direct IO read/write, the number of requests fuse consumed is limited by the number of such processes (actually their threads). In contrast, if userspace requests 1GB direct IO read/write, it would be unreasonable to issue 1GB/128K==8192 fuse requests simultaneously. That's wheremax_backgroundsteps in.Quoted 4 lines
Ah, that makes sense. Are these two cases meant as examples, or is that an exhaustive list? Because I would have thought that other cases should be writing of cached data (when writeback caching is enabled), and asynchronous I/O from userspace...?
I think that's exhaustive list, but I can miss something.
As for writing of cached data, that definitely doesn't go through background requests. Here we rely on flusher: fuse will allocate as many requests as the flusher wants to writeback.
Quoted 5 lines
Quoted 3 lines
Buffered AIO READs actually block in
submit_iountil fully processed. So it's just another example of "natural" limitation I told above.Not sure I understand. What is it that's blocking? It can't be the userspace process, because then it wouldn't be asynchronous I/O...
Surprise! Alas, Linux kernel does NOT process buffered AIO reads in async manner. You can verify it yourself by strace-ing a simple program looping over
io_submit+io_getevents: for direct IO (as expected)io_submitreturns immediately whileio_geteventswaits for actual IO; in contrast, for buffered IO (surprisingly)io_submitwaits for actual IO whileio_geteventsreturns immediately. Presumably, people are supposed to use mmap-ed read/writes rather than buffered AIO.
Quoted 7 lines
Quoted 4 lines
What about buffered, asynchronous writes when writeback cache is disabled? It sounds as if
io_submitdoes not block (so userspace could create an unlimited number), nor can the kernel coalesce them (since writeback caching is disabled).I've never looked closely at it. Do you have a particular use case or concern?
My only concern is to accurately describe the effects of the "max_background" parameter in the libfuse documentation.
At the moment most FUSE filesystems don't use writeback caching (because there is no stable libfuse release out that supports it). On the other hand, most filesystem are probably also not too worried about the behavior when userspace submits a large number of asynchronous write requests. But I think it would still be important to correctly describe this case. If io_submit does not block, and the request does not count as a background request, wouldn't this be a bug that should be fixed? Or is there anything else that would limit the number of such requests?
Best,
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«