net: introduce helper sendpage_ok() in include/linux/net.h
Subsystems (6)
PATCH v5 1/3 net: introduce helper sendpage_ok() in include/linux/net.h
The original problem was from nvme-over-tcp code, who mistakenly uses kernel_sendpage() to send pages allocated by __get_free_pages() without __GFP_COMP flag. Such pages don't have refcount (page_count is 0) on tail pages, sending them by kernel_sendpage() may trigger a kernel panic from a corrupted kernel heap, because these pages are incorrectly freed in network stack as page_count 0 pages.
This patch introduces a helper sendpage_ok(), it returns true if the checking page,
- is not slab page: PageSlab(page) is false.
- has page refcount: page_count(page) is not zero
All drivers who want to send page to remote end by kernel_sendpage() may use this helper to check whether the page is OK. If the helper does not return true, the driver should try other non sendpage method (e.g. sock_no_sendpage()) to handle the page.
---
v5, include linux/mm.h in include/linux/net.h
v4, change sendpage_ok() as an inline helper, and post it as
separate patch.
v3, introduce a more common sendpage_ok()
v2, fix typo in patch subject
v1, the initial version.
include/linux/net.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
include/linux/net.h +16 -0
@@ -21,6 +21,7 @@#include <linux/rcupdate.h> #include <linux/once.h> #include <linux/fs.h> +#include <linux/mm.h> #include <linux/sockptr.h> #include <uapi/linux/net.h>
@@ -286,6 +287,21 @@ do { \#define net_get_random_once_wait (buf, nbytes) \ get_random_once_wait((buf), (nbytes)) +/* +* E.g. XFS meta- & log-data is in slab pages, or bcache meta +* data pages, or other high order pages allocated by +* __get_free_pages() without __GFP_COMP, which have a page_count +* of 0 and/or have PageSlab() set. We cannot use send_page for +* those, as that does get_page(); put_page(); and would cause +* either a VM_BUG directly, or __page_cache_release a page that +* would actually still be referenced by someone, leading to some +* obscure delayed Oops somewhere else. +*/ +static inline bool sendpage_ok (struct page *page) +{ +return (!PageSlab (page) &&page_count (page) >=1 ); +} +int kernel_sendmsg (struct socket *sock,struct msghdr *msg,struct kvec *vec,size_t num,size_t len); int kernel_sendmsg_locked(struct sock *sk, struct msghdr *msg,
Signature · 1 line
2.26.2
PATCH v5 2/3 nvme-tcp: check page by sendpage_ok() before calling kernel_sendpage()
Currently nvme_tcp_try_send_data() doesn't use kernel_sendpage() to send slab pages. But for pages allocated by __get_free_pages() without __GFP_COMP, which also have refcount as 0, they are still sent by kernel_sendpage() to remote end, this is problematic.
The new introduced helper sendpage_ok() checks both PageSlab tag and page_count counter, and returns true if the checking page is OK to be sent by kernel_sendpage().
This patch fixes the page checking issue of nvme_tcp_try_send_data() with sendpage_ok(). If sendpage_ok() returns true, send this page by kernel_sendpage(), otherwise use sock_no_sendpage to handle this page.
--- drivers/nvme/host/tcp.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +3 -4
@@ -912,12 +912,11 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)else flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; -/* can't zcopy slab pages */ -if (unlikely (PageSlab (page))) { - ret =sock_no_sendpage (queue->sock, page, offset, len, +if (sendpage_ok (page)) { + ret =kernel_sendpage (queue->sock, page, offset, len, flags); }else { - ret =kernel_sendpage (queue->sock, page, offset, len, + ret =sock_no_sendpage (queue->sock, page, offset, len, flags); }if (ret <=0 )
Signature · 1 line
2.26.2
PATCH v5 3/3 drbd: code cleanup by using sendpage_ok() to check page for kernel_sendpage()
In reply to Coly LiIn _drbd_send_page() a page is checked by following code before sending it by kernel_sendpage(),
(page_count(page) < 1) || PageSlab(page)
If the check is true, this page won't be send by kernel_sendpage() and handled by sock_no_sendpage().
This kind of check is exactly what macro sendpage_ok() does, which is introduced into include/linux/net.h to solve a similar send page issue in nvme-tcp code.
This patch uses macro sendpage_ok() to replace the open coded checks to page type and refcount in _drbd_send_page(), as a code cleanup.
--- drivers/block/drbd/drbd_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/drbd/drbd_main.c +1 -1
@@ -1553,7 +1553,7 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa* put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ -if (drbd_disable_sendpage || (page_count (page) <1 ) ||PageSlab (page)) +if (drbd_disable_sendpage || !sendpage_ok (page))return _drbd_no_send_page (peer_device, page, offset, size, msg_flags); msg_flags |=MSG_NOSIGNAL ;
Signature · 1 line
2.26.2
Quoted 40 lines · Coly Li
The original problem was from nvme-over-tcp code, who mistakenly uses
kernel_sendpage()to send pages allocated by__get_free_pages()without__GFP_COMPflag. Such pages don't have refcount (page_countis 0) on tail pages, sending them bykernel_sendpage()may trigger a kernel panic from a corrupted kernel heap, because these pages are incorrectly freed in network stack aspage_count0 pages.This patch introduces a helper
sendpage_ok(), it returns true if the checking page,
- is not slab page: PageSlab(page) is false.
- has page refcount:page_count(page)is not zeroAll drivers who want to send page to remote end by
kernel_sendpage()may use this helper to check whether the page is OK. If the helper does not return true, the driver should try other non sendpage method (e.g.sock_no_sendpage()) to handle the page.include/linux/net.h +6
-0@@ -21,6 +21,7 @@#include <linux/rcupdate.h> #include <linux/once.h> #include <linux/fs.h> +#include <linux/mm.h> #include <linux/sockptr.h> #include <uapi/linux/net.h> @@ -286,6 +287,21 @@ do { \#define net_get_random_once_wait (buf, nbytes) \ get_random_once_wait((buf), (nbytes)) +/* +* E.g. XFS meta- & log-data is in slab pages, or bcache meta +* data pages, or other high order pages allocated by +* __get_free_pages() without __GFP_COMP, which have a page_count +* of 0 and/or have PageSlab() set. We cannot use send_page for
include/linux/net.h do { \ · line 304
Quoted 10 lines
include/linux/net.h +9
-0+* those, as that does get_page(); put_page(); and would cause +* either a VM_BUG directly, or __page_cache_release a page that +* would actually still be referenced by someone, leading to some +* obscure delayed Oops somewhere else. +*/ +static inline bool sendpage_ok (struct page *page) +{ +return (!PageSlab (page) &&page_count (page) >=1 ); +}
return (A);
Can simply be written :
return A;
In this case :
return !PageSlab(page) && page_count(page) >= 1;
BTW, do you have plans to refine code added with commit a10674bf2406afc2554f9c7d31b2dc65d6a27fd9 ("tcp: detecting the misuse of .sendpage for Slab objects")
Quoted 38 lines · Eric Dumazet
Quoted 35 lines · Coly Li
The original problem was from nvme-over-tcp code, who mistakenly uses
kernel_sendpage()to send pages allocated by__get_free_pages()without__GFP_COMPflag. Such pages don't have refcount (page_countis 0) on tail pages, sending them bykernel_sendpage()may trigger a kernel panic from a corrupted kernel heap, because these pages are incorrectly freed in network stack aspage_count0 pages.This patch introduces a helper
sendpage_ok(), it returns true if the checking page,
- is not slab page: PageSlab(page) is false.
- has page refcount:page_count(page)is not zeroAll drivers who want to send page to remote end by
kernel_sendpage()may use this helper to check whether the page is OK. If the helper does not return true, the driver should try other non sendpage method (e.g.sock_no_sendpage()) to handle the page.include/linux/net.h +1
-0@@ -21,6 +21,7 @@#include <linux/rcupdate.h> #include <linux/once.h> #include <linux/fs.h> +#include <linux/mm.h> #include <linux/sockptr.h> #include <uapi/linux/net.h> @@ -286,6 +287,21 @@ do { \#define net_get_random_once_wait (buf, nbytes) \ get_random_once_wait((buf), (nbytes))
Quoted 15 lines
Quoted 5 lines
include/linux/net.h +5
-0+/* +* E.g. XFS meta- & log-data is in slab pages, or bcache meta +* data pages, or other high order pages allocated by +* __get_free_pages() without __GFP_COMP, which have a page_count +* of 0 and/or have PageSlab() set. We cannot use send_page for include/linux/net.h do { \
Quoted 10 lines
include/linux/net.h +9
-0+* those, as that does get_page(); put_page(); and would cause +* either a VM_BUG directly, or __page_cache_release a page that +* would actually still be referenced by someone, leading to some +* obscure delayed Oops somewhere else. +*/ +static inline bool sendpage_ok (struct page *page) +{ +return (!PageSlab (page) &&page_count (page) >=1 ); +}
Hi Eric,
Quoted 10 lines
return (A);
Can simply be written :
return A;
In this case :
return !PageSlab(page) &&
page_count(page)>= 1;
Sure, I update it in v6 series.
Quoted 4 lines
BTW, do you have plans to refine code added with commit
a10674bf2406afc2554f9c7d31b2dc65d6a27fd9("tcp: detecting the misuse of .sendpage for Slab objects")
Thanks for the hint, I will remove this piece in v6 series.
Coly Li