sendfile to nonblocking socket
Hello
I'm testing a web server nginx for films sharing in my LAN. And I've got some interesting results. When I tried to download film or another big file via gigabit link, I've got sendfile block with nonblocking socket. Strace log in attach. Some commens
#enabling nonblock on fd 3
20:51:20 ioctl(3, FIONBIO, [1]) = 0
#normal nonblocking sendfile, asked 2147480274 bytes, sent 236164 bytes, so nonblocking works 20:51:20 sendfile(3, 8, [847150], 2147480274) = 236164
#sendfile 390 M, 6 seconds
20:51:22 sendfile(3, 8, [102578266], 2147481510) = 390115144
#sendfile 1000 M, 15 seconds
20:51:40 sendfile(3, 8, [1303409692], 2147482596) = 1008100764
#sendfile ~2G, 30 seconds
20:51:55 sendfile(3, 8, [2312288408], 1982678888) = 1982678888
As I see, nonblocking mode is enabled - sendfile sends less than asked. But 2G via single 30 seconds sendfile call - this is blocking call. How can I avoid that? I prefer sendfile as fastest way to send file content to network socket. The problem with sendfile block on nonblocking socket has place only when I'm using network connection, that is faster than my hard disk, for example gigabit NIC or localhost. When I use 100Mbit NIC, which is slower, than my hard disk, I got small and fast sendfile calls without blocking.
My kernel is Linux 2.6.20.3-grsec x86_64. I verified that also on 2.6.18 - same results. Please advise.
ps: I've did same tests with lighttpd's sendfile and got same results - block on nonblocking socket, when network is faster than disk.
Thank you,
Alex
strace -tp 10305
Process 10305 attached - interrupt to quit
20:51:17 write(13, "2007/04/23 20:51:17 [info] 10305"..., 85) = 85
20:51:17 epoll_wait(7, {{EPOLLIN, {u32=1054142480, u64=50471214837776}}}, 512, 4294967295) = 1
20:51:20 accept(10, {sa_family=AF_INET, sin_port=htons(55446), sin_addr=inet_addr("192.168.78.1")}, [5056729052170158096]) = 3
20:51:20 ioctl(3, FIONBIO, [1]) = 0
20:51:20 epoll_ctl(7, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLET, {u32=1054142784, u64=50471214838080}}) = 0 20:51:20 epoll_wait(7, {{EPOLLIN, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:20 recvfrom(3, "GET /3.tmp HTTP/1.0\r\nUser-Agent:"..., 1024, 0, NULL, NULL) = 147
20:51:20 open("/var/www/cacti/htdocs/3.tmp", O_RDONLY) = 8
20:51:20 fstat(8, {st_mode=S_IFREG|0644, st_size=4294967296, ...}) = 0
20:51:20 setsockopt(3, SOL_TCP, TCP_CORK, [1], 4) = 0
20:51:20 writev(3, [{"HTTP/1.1 200 OK\r\nServer: nginx/0"..., 262}], 1) = 262
20:51:20 sendfile(3, 8, [0], 2147479552) = 20576
20:51:20 epoll_ctl(7, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=1054142784, u64=50471214838080}}) = 0 20:51:20 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:20 sendfile(3, 8, [20576], 2147483552) = 55568
20:51:20 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599981) = 1
20:51:20 sendfile(3, 8, [76144], 2147481232) = 27784
20:51:20 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599979) = 1
20:51:20 sendfile(3, 8, [103928], 2147482120) = 90298
20:51:20 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599979) = 1
20:51:20 sendfile(3, 8, [194226], 2147481934) = 652924
20:51:20 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599978) = 1
20:51:20 sendfile(3, 8, [847150], 2147480274) = 236164
20:51:20 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599970) = 1
20:51:21 sendfile(3, 8, [1083314], 2147481678) = 4549630
20:51:21 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599960) = 1
20:51:21 sendfile(3, 8, [5632944], 2147482704) = 38529462
20:51:21 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599894) = 1
20:51:21 sendfile(3, 8, [44162406], 2147480218) = 33451936
20:51:22 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:22 sendfile(3, 8, [77614342], 2147480314) = 20393456
20:51:22 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:22 sendfile(3, 8, [98007798], 2147480842) = 583464
20:51:22 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:22 sendfile(3, 8, [98591262], 2147483106) = 687654
20:51:22 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599959) = 1
20:51:22 sendfile(3, 8, [99278916], 2147483580) = 798790
20:51:22 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599945) = 1
20:51:22 sendfile(3, 8, [100077706], 2147483510) = 2500560
20:51:22 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599931) = 1
20:51:22 sendfile(3, 8, [102578266], 2147481510) = 390115144
20:51:28 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599910) = 1
20:51:28 sendfile(3, 8, [492693410], 2147481694) = 1007170
20:51:28 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:28 sendfile(3, 8, [493700580], 2147482140) = 1028008
20:51:28 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599984) = 1
20:51:28 sendfile(3, 8, [494728588], 2147482228) = 804430152
20:51:40 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599971) = 1
20:51:40 sendfile(3, 8, [1299158740], 2147481900) = 1069684
20:51:40 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:40 sendfile(3, 8, [1300228424], 2147481272) = 1062738
20:51:40 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599976) = 1
20:51:40 sendfile(3, 8, [1301291162], 2147483494) = 1007170
20:51:40 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599954) = 1
20:51:40 sendfile(3, 8, [1302298332], 2147479844) = 1111360
20:51:40 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599931) = 1
20:51:40 sendfile(3, 8, [1303409692], 2147482596) = 1008100764
20:51:55 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 599905) = 1
20:51:55 sendfile(3, 8, [2311510456], 1983456840) = 777952
20:51:55 epoll_wait(7, {{EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 600000) = 1
20:51:55 sendfile(3, 8, [2312288408], 1982678888) = 1982678888
20:52:25 write(16, "192.168.78.1 - voron [23/Apr/200"..., 114) = 114
20:52:25 close(8) = 0
20:52:25 setsockopt(3, SOL_TCP, TCP_CORK, [0], 4) = 0
20:52:25 recvfrom(3, 0x57d880, 1024, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
20:52:25 epoll_wait(7, {{EPOLLIN|EPOLLOUT, {u32=1054142784, u64=50471214838080}}}, 512, 75000) = 1 20:52:25 recvfrom(3, "", 1024, 0, NULL, NULL) = 0
20:52:25 close(3)
From: voron <[email protected]>
Date: Tue, 24 Apr 2007 00:13:27 +0300
Quoted 1 line
As I see, nonblocking mode is enabled - sendfile sends less than asked.
The socket is marked as non-blocking, but the disk I/O is not.
It's blocking on the disk I/O not the socket part of the operation.
Quoted 10 lines
As I see, nonblocking mode is enabled - sendfile sends less than asked. But 2G via single 30 seconds sendfile call - this is blocking call. How can I avoid that? I prefer sendfile as fastest way to send file content to network socket. The problem with sendfile block on nonblocking socket has place only when I'm using network connection, that is faster than my hard disk, for example gigabit NIC or localhost. When I use 100Mbit NIC, which is slower, than my hard disk, I got small and fast sendfile calls without blocking.
Quoted 2 lines
My kernel is Linux 2.6.20.3-grsec
x86_64. I verified that also on 2.6.18 - same results. Please advise.
You have a misunderstanding about the semantics of 'sendfile'. The 'sendfile' function is just a more efficient version of a read followed by a write. If you did a read followed by a write, it would block as well (in the read).
DS
Quoted 4 lines · David Miller
From: voron <[email protected]>
Date: Tue, 24 Apr 2007 00:13:27 +0300
Quoted 9 lines
Quoted 2 lines
As I see, nonblocking mode is enabled - sendfile sends less than asked.
The socket is marked as non-blocking, but the disk I/O is not.
It's blocking on the disk I/O not the socket part of the operation.
How can I told kernel to not block on disk I/O? I tried non-blocking on disk i/o fd, but it seems to be ignored. Strace with both nonblocing disk fd and socket fd in attach
#non-blocking on socket fd enabled
04:34:07 ioctl(9, FIONBIO, [1]) = 0
#non-blocking on disk fd enabled
04:34:11 ioctl(12, FIONBIO, [1]) = 0
#normal sendfile
04:34:07 sendfile(9, 12, [444282], 2147477638) = 812682
#32 seconds sendfile
04:34:11 sendfile(9, 12, [261474962], 2147476846) = 2144612230
Thank you,
Alex
04:34:04 write(3, "2007/04/24 04:34:04 [info] 32390"..., 85) = 85
04:34:04 epoll_wait(11, {{EPOLLIN, {u32=3618562064, u64=61614924423184}}}, 512, 4294967295) = 1
04:34:07 accept(8, {sa_family=AF_INET, sin_port=htons(52673), sin_addr=inet_addr("192.168.78.1")}, [5056848310527066128]) = 9
04:34:07 ioctl(9, FIONBIO, [1]) = 0
04:34:07 epoll_ctl(11, EPOLL_CTL_ADD, 9, {EPOLLIN|EPOLLET, {u32=3618562369, u64=61614924423489}}) = 0 04:34:07 epoll_wait(11, {{EPOLLIN, {u32=3618562369, u64=61614924423489}}}, 512, 600000) = 1
04:34:07 recvfrom(9, "GET /3.tmp HTTP/1.0\r\nUser-Agent:"..., 1024, 0, NULL, NULL) = 147
04:34:07 open("/var/www/cacti/htdocs/3.tmp", O_RDONLY) = 12
04:34:07 fstat(12, {st_mode=S_IFREG|0644, st_size=4294967296, ...}) = 0
04:34:07 setsockopt(9, SOL_TCP, TCP_CORK, [1], 4) = 0
04:34:07 writev(9, [{"HTTP/1.1 200 OK\r\nServer: nginx/0"..., 262}], 1) = 262
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [0], 2147475456) = 20576
04:34:07 epoll_ctl(11, EPOLL_CTL_MOD, 9, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=3618562369, u64=61614924423489}}) = 0 04:34:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 600000) = 1
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [20576], 2147479456) = 55568
04:34:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599975) = 1
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [76144], 2147477136) = 41676
04:34:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599973) = 1
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [117820], 2147476420) = 83352
04:34:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599972) = 1
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [201172], 2147479084) = 243110
04:34:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599971) = 1
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [444282], 2147477638) = 812682
04:34:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599968) = 1
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [1256964], 2147475964) = 7501680
04:34:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599950) = 1
04:34:07 ioctl(12, FIONBIO, [1]) = 0
04:34:07 sendfile(9, 12, [8758644], 2147478156) = 234274688
04:34:10 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599846) = 1
04:34:10 ioctl(12, FIONBIO, [1]) = 0
04:34:10 sendfile(9, 12, [243033332], 2147478284) = 583464
04:34:10 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 600000) = 1
04:34:10 ioctl(12, FIONBIO, [1]) = 0
04:34:10 sendfile(9, 12, [243616796], 2147476452) = 16712076
04:34:11 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599990) = 1
04:34:11 ioctl(12, FIONBIO, [1]) = 0
04:34:11 sendfile(9, 12, [260328872], 2147476056) = 1146090
04:34:11 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599766) = 1
04:34:11 ioctl(12, FIONBIO, [1]) = 0
04:34:11 sendfile(9, 12, [261474962], 2147476846) = 2144612230
04:34:43 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599754) = 1
04:34:43 ioctl(12, FIONBIO, [1]) = 0
04:34:43 sendfile(9, 12, [2406087192], 1888880104) = 611248
04:34:43 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 600000) = 1
04:34:43 ioctl(12, FIONBIO, [1]) = 0
04:34:43 sendfile(9, 12, [2406698440], 1888268856) = 1000890816
04:34:57 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599991) = 1
04:34:57 ioctl(12, FIONBIO, [1]) = 0
04:34:57 sendfile(9, 12, [3407589256], 887378040) = 972440
04:34:58 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 600000) = 1
04:34:58 ioctl(12, FIONBIO, [1]) = 0
04:34:58 sendfile(9, 12, [3408561696], 886405600) = 861304
04:34:58 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599977) = 1
04:34:58 ioctl(12, FIONBIO, [1]) = 0
04:34:58 sendfile(9, 12, [3409423000], 885544296) = 993278
04:34:58 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599954) = 1
04:34:58 ioctl(12, FIONBIO, [1]) = 0
04:34:58 sendfile(9, 12, [3410416278], 884551018) = 847412
04:34:58 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599930) = 1
04:34:58 ioctl(12, FIONBIO, [1]) = 0
04:34:58 sendfile(9, 12, [3411263690], 883703606) = 4021734
04:34:58 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599908) = 1
04:34:58 ioctl(12, FIONBIO, [1]) = 0
04:34:58 sendfile(9, 12, [3415285424], 879681872) = 492166998
04:35:07 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 599874) = 1
04:35:07 ioctl(12, FIONBIO, [1]) = 0
04:35:07 sendfile(9, 12, [3907452422], 387514874) = 104002458
04:35:09 epoll_wait(11, {{EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 600000) = 1
04:35:09 ioctl(12, FIONBIO, [1]) = 0
04:35:09 sendfile(9, 12, [4011454880], 283512416) = 283512416
04:35:15 write(6, "192.168.78.1 - voron [24/Apr/200"..., 114) = 114
04:35:15 close(12) = 0
04:35:15 setsockopt(9, SOL_TCP, TCP_CORK, [0], 4) = 0
04:35:15 recvfrom(9, 0x5804d0, 1024, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
04:35:15 epoll_wait(11, {{EPOLLIN|EPOLLOUT, {u32=3618562369, u64=61614924423489}}}, 512, 75000) = 1 04:35:15 recvfrom(9, "", 1024, 0, NULL, NULL) = 0
04:35:15 close(9) = 0
04:35:15 epoll_wait(11,
David Schwartz пишет:
Quoted 4 lines
You have a misunderstanding about the semantics of 'sendfile'. The 'sendfile' function is just a more efficient version of a read followed by a write. If you did a read followed by a write, it would block as well (in the read).
DS
sendfile function is not just a more efficient version of a read followed by a write. It reads from one fd and write to another at tha same time. Please try to read 2G, and then write 2G - and how much memory you will be need and how much time you will loose while reading 2G from disk, but not writing them to socket. If you know more efficient method to transfer file from disk to network - please advise. Now all I want is really non-blocking sendfile. Currently sendfile is non-blocking on network, but not on disk i/o. And when I have network faster than disk - I get block.
Thank you,
Alex
Quoted 7 lines
David Schwartz пишет:
Quoted 1 line
You have a misunderstanding about the semantics of 'sendfile'.
The 'sendfile' function is just a more efficient version of a read followed by a write. If you did a read followed by a write, it would block as well (in the read).
Quoted 1 line
Quoted 7 lines
Quoted 1 line
DS
sendfile function is not just a more efficient version of a read followed by a write. It reads from one fd and write to another at tha same time. Please try to read 2G, and then write 2G - and how much memory you will be need and how much time you will loose while reading 2G from disk, but not writing them to socket.
You are correct. What I meant to say was that it's just a more efficient version of 'mmap'ing a file and then 'write'ing from the 'mmap'. The 'write' to a non-blocking socket can still 'block' on disk I/O.
Quoted 5 lines
If you know more
efficient method to transfer file from disk to network - please advise. Now all I want is really non-blocking sendfile. Currently sendfile is non-blocking on network, but not on disk i/o. And when I have network faster than disk - I get block.
There are many different techniques and which is correct depends on what direction you want to go. POSIX asynchronous I/O is one possibility. Threads plus epoll is another. It really depends upon how much performance you need, how much complexity you can tolerate, and how portable you need to be.
DS
Hello David,
Quoted 5 lines · you
Quoted 5 lines
sendfile function is not just a more efficient version of a read followed by a write. It reads from one fd and write to another at tha same time. Please try to read 2G, and then write 2G - and how much memory you will be need and how much time you will loose while reading 2G from disk, but not writing them to socket.
DS> You are correct. What I meant to say was that it's just a DS> more efficient version of 'mmap'ing a file and then 'write'ing DS> from the 'mmap'. The 'write' to a non-blocking socket can still DS> 'block' on disk I/O.
How can I avoid that blocking? Or maybe another question - how can I deliver data from disk to network with minimal copy operations, etc.
Quoted 5 lines
Quoted 5 lines
If you know more
efficient method to transfer file from disk to network - please advise. Now all I want is really non-blocking sendfile. Currently sendfile is non-blocking on network, but not on disk i/o. And when I have network faster than disk - I get block.
DS> There are many different techniques and which is correct
DS> depends on what direction you want to go.
_very_ fast frontend web-server :) using as much kernel features as possible DS> POSIX asynchronous I/O is one possibility.
aio does not support direct transfer file->socket. Using some hints, like aio reading into shared memory and then sendfile from shared memory, like lighttpd does, is not what I want. My tests showing
that aio-sendfile realization in lighttpd is slower than sendfile.
I think, sendfile uses less copy operations(maybe even zerocopy), than current aio realizations in kernel.
DS> Threads plus epoll is another.
20k threads and maybe more is too much :). Look at http://nginx.net/
senction "Architecture and scalability" for example.
DS> It really depends upon how much performance you need
all, that hardware can take and hold :)
Signature · 2 lines
Best regards, Alex mailto:[email protected]
Quoted 10 lines · On Tue, 24 Apr 2007 14:33:48 +0400 Alex Vorona <[email protected]> wrote:
Hello David,
Quoted 5 lines · you
Quoted 5 lines
sendfile function is not just a more efficient version of a read followed by a write. It reads from one fd and write to another at tha same time. Please try to read 2G, and then write 2G - and how much memory you will be need and how much time you will loose while reading 2G from disk, but not writing them to socket.
Quoted 8 lines
DS> You are correct. What I meant to say was that it's just a DS> more efficient version of 'mmap'ing a file and then 'write'ing DS> from the 'mmap'. The 'write' to a non-blocking socket can still DS> 'block' on disk I/O.
How can I avoid that blocking? Or maybe another question - how can I deliver data from disk to network with minimal copy operations, etc.
I believe the modern way would be to use splice() system call :
loop
splice(from disk , to pipe) with SPLICE_F_NONBLOCK wait_event_from_epoll (pipe ready for reading) splice(from pipe, to socket) wait_event_from_epoll (socket ready for writing)
But :
1) I am not sure epoll can get an event when page(s) is/are ready in pipe (ie disk delivered the page(s) to cache)
2) I am not sure splice() to socket is actually implemented with 0-copy
3) I am not sure pipe capacity (16 pages) would be enough to get a good readahead.
Eric
Quoted 7 lines
DS> Threads plus epoll is another.
20k threads and maybe more is too much :). Look at http://nginx.net/
senction "Architecture and scalability" for example.
DS> It really depends upon how much performance you need all, that hardware can take and hold :)
Why would you want 20k threads? You aren't seriously suggesting that you need to have 20,000 outstanding disk operations, are you? Surely you don't think that would be efficient. If the disk is the limiting factor, it may get slightly faster as you pend more concurrent requests, but surely 20,000 is not the best number! (256 is probably closer to the optimal value, and it may be less.)
Your application has to manage the outstanding disk read requests. I don't know of any way to foist this task on the kernel. Perhaps a pool of disk read threads?
I would keep a flag for each connection to track whether the last write got a 'would block' or was incomplete. So long as this flag is clear, let the disk read thread attempt the socket 'write'. If the disk read thread gets a partial write (or a would block indication), set the flag on the socket and let the socket I/O threads takeover the connection (based on 'epoll' notification). When a write completes and you need more disk data, clear the flag and let the disk read threads takeover the connection until a write blocks again. (This disk read threads can use 'sendfile' or 'splice' so long as they don't block on the socket.)
Perhaps the disk read threads should be using 'mmap' with MAP_POPULATE.
There are certainly many possible approaches.
DS