Introduce sendpage_ok() to detect misused sendpage in network related drivers
Subsystems (10)
v6 cover Introduce sendpage_ok() to detect misused sendpage in network related drivers
This series was original by a bug fix in nvme-over-tcp driver which only checked whether a page was allocated from slab allcoator, but forgot to check its page_count: The page handled by sendpage should be neither a Slab page nor 0 page_count page.
As Sagi Grimberg suggested, the original fix is refind to a more common inline routine:
static inline bool sendpage_ok(struct page *page)
{
return (!PageSlab(page) && page_count(page) >= 1);
}If sendpage_ok() returns true, the checking page can be handled by the zero copy sendpage method in network layer.
The first patch in this series introduces sendpage_ok() in header file include/linux/net.h, the second patch fixes the page checking issue in nvme-over-tcp driver, the third patch adds page_count check by using sendpage_ok() in do_tcp_sendpages() as Eric Dumazet suggested, and all rested patches just replace existing open coded checks with the inline sendpage_ok() routine.
Coly Li
---
Changelog:
v6: fix page check in do_tcp_sendpages(), and replace other open coded
checks with sendpage_ok() in libceph, iscsi drivers.
v5, include linux/mm.h in include/linux/net.h
v4, change sendpage_ok() as an inline helper, and post it as
separate patch.
v3, introduce a more common sendpage_ok()
v2, fix typo in patch subject
v1, the initial version.
Coly Li (6):
net: introduce helper sendpage_ok() in include/linux/net.h
nvme-tcp: check page by sendpage_ok() before calling kernel_sendpage()
tcp: use sendpage_ok() to detect misused .sendpage
drbd: code cleanup by using sendpage_ok() to check page for
kernel_sendpage()
scsi: libiscsi: use sendpage_ok() in iscsi_tcp_segment_map()
libceph: use sendpage_ok() in ceph_tcp_sendpage()
drivers/block/drbd/drbd_main.c | 2 +- drivers/nvme/host/tcp.c | 7 +++---- drivers/scsi/libiscsi_tcp.c | 2 +- include/linux/net.h | 16 ++++++++++++++++ net/ceph/messenger.c | 2 +- net/ipv4/tcp.c | 3 ++- 6 files changed, 24 insertions(+), 8 deletions(-)
Signature · 1 line
2.26.2
PATCH v6 1/6 net: introduce helper sendpage_ok() in include/linux/net.h
The original problem was from nvme-over-tcp code, who mistakenly uses kernel_sendpage() to send pages allocated by __get_free_pages() without __GFP_COMP flag. Such pages don't have refcount (page_count is 0) on tail pages, sending them by kernel_sendpage() may trigger a kernel panic from a corrupted kernel heap, because these pages are incorrectly freed in network stack as page_count 0 pages.
This patch introduces a helper sendpage_ok(), it returns true if the checking page,
- is not slab page: PageSlab(page) is false.
- has page refcount: page_count(page) is not zero
All drivers who want to send page to remote end by kernel_sendpage() may use this helper to check whether the page is OK. If the helper does not return true, the driver should try other non sendpage method (e.g. sock_no_sendpage()) to handle the page.
--- include/linux/net.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
include/linux/net.h +16 -0
@@ -21,6 +21,7 @@#include <linux/rcupdate.h> #include <linux/once.h> #include <linux/fs.h> +#include <linux/mm.h> #include <linux/sockptr.h> #include <uapi/linux/net.h>
@@ -286,6 +287,21 @@ do { \#define net_get_random_once_wait (buf, nbytes) \ get_random_once_wait((buf), (nbytes)) +/* +* E.g. XFS meta- & log-data is in slab pages, or bcache meta +* data pages, or other high order pages allocated by +* __get_free_pages() without __GFP_COMP, which have a page_count +* of 0 and/or have PageSlab() set. We cannot use send_page for +* those, as that does get_page(); put_page(); and would cause +* either a VM_BUG directly, or __page_cache_release a page that +* would actually still be referenced by someone, leading to some +* obscure delayed Oops somewhere else. +*/ +static inline bool sendpage_ok (struct page *page) +{ +return (!PageSlab (page) &&page_count (page) >=1 ); +} +int kernel_sendmsg (struct socket *sock,struct msghdr *msg,struct kvec *vec,size_t num,size_t len); int kernel_sendmsg_locked(struct sock *sk, struct msghdr *msg,
Signature · 7 lines
2.26.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v6 2/6 nvme-tcp: check page by sendpage_ok() before calling kernel_sendpage()
In reply to Coly LiCurrently nvme_tcp_try_send_data() doesn't use kernel_sendpage() to send slab pages. But for pages allocated by __get_free_pages() without __GFP_COMP, which also have refcount as 0, they are still sent by kernel_sendpage() to remote end, this is problematic.
The new introduced helper sendpage_ok() checks both PageSlab tag and page_count counter, and returns true if the checking page is OK to be sent by kernel_sendpage().
This patch fixes the page checking issue of nvme_tcp_try_send_data() with sendpage_ok(). If sendpage_ok() returns true, send this page by kernel_sendpage(), otherwise use sock_no_sendpage to handle this page.
--- drivers/nvme/host/tcp.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +3 -4
@@ -912,12 +912,11 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)else flags |=MSG_MORE |MSG_SENDPAGE_NOTLAST ; -/* can't zcopy slab pages */ -if (unlikely (PageSlab (page))) { - ret =sock_no_sendpage (queue->sock, page, offset, len, +if (sendpage_ok (page)) { + ret =kernel_sendpage (queue->sock, page, offset, len, flags); }else { - ret =kernel_sendpage (queue->sock, page, offset, len, + ret =sock_no_sendpage (queue->sock, page, offset, len, flags); }if (ret <=0 )
Signature · 7 lines
2.26.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v6 3/6 tcp: use sendpage_ok() to detect misused .sendpage
In reply to Coly Licommit a10674bf2406 ("tcp: detecting the misuse of .sendpage for Slab objects") adds the checks for Slab pages, but the pages don't have page_count are still missing from the check.
Network layer's sendpage method is not designed to send page_count 0 pages neither, therefore both PageSlab() and page_count() should be both checked for the sending page. This is exactly what sendpage_ok() does.
This patch uses sendpage_ok() in do_tcp_sendpages() to detect misused .sendpage, to make the code more robust.
--- net/ipv4/tcp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
net/ipv4/tcp.c +2 -1
@@ -970,7 +970,8 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset,long timeo =sock_sndtimeo (sk, flags &MSG_DONTWAIT );if (IS_ENABLED (CONFIG_DEBUG_VM ) && -WARN_ONCE (PageSlab (page),"page must not be a Slab one" )) +WARN_ONCE (!sendpage_ok (page)), +"page must not be a Slab one and have page_count > 0" )return -EINVAL ;/* Wait for a connection to finish. One exception is TCP Fast Open
Signature · 1 line
2.26.2
PATCH v6 4/6 drbd: code cleanup by using sendpage_ok() to check page for kernel_sendpage()
In reply to Coly LiIn _drbd_send_page() a page is checked by following code before sending it by kernel_sendpage(),
(page_count(page) < 1) || PageSlab(page)
If the check is true, this page won't be send by kernel_sendpage() and handled by sock_no_sendpage().
This kind of check is exactly what macro sendpage_ok() does, which is introduced into include/linux/net.h to solve a similar send page issue in nvme-tcp code.
This patch uses macro sendpage_ok() to replace the open coded checks to page type and refcount in _drbd_send_page(), as a code cleanup.
--- drivers/block/drbd/drbd_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/drbd/drbd_main.c +1 -1
@@ -1553,7 +1553,7 @@ static int _drbd_send_page(struct drbd_peer_device *peer_device, struct page *pa* put_page(); and would cause either a VM_BUG directly, or * __page_cache_release a page that would actually still be referenced * by someone, leading to some obscure delayed Oops somewhere else. */ -if (drbd_disable_sendpage || (page_count (page) <1 ) ||PageSlab (page)) +if (drbd_disable_sendpage || !sendpage_ok (page))return _drbd_no_send_page (peer_device, page, offset, size, msg_flags); msg_flags |=MSG_NOSIGNAL ;
Signature · 1 line
2.26.2
PATCH v6 5/6 scsi: libiscsi: use sendpage_ok() in iscsi_tcp_segment_map()
In reply to Coly LiIn iscsci driver, iscsi_tcp_segment_map() uses the following code to check whether the page should or not be handled by sendpage:
if (!recv && page_count(sg_page(sg)) >= 1 && !PageSlab(sg_page(sg)))
The "page_count(sg_page(sg)) >= 1 && !PageSlab(sg_page(sg)" part is to make sure the page can be sent to network layer's zero copy path. This part is exactly what sendpage_ok() does.
This patch uses use sendpage_ok() in iscsi_tcp_segment_map() to replace the original open coded checks.
--- drivers/scsi/libiscsi_tcp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/scsi/libiscsi_tcp.c +1 -1
@@ -128,7 +128,7 @@ static void iscsi_tcp_segment_map(struct iscsi_segment *segment, int recv)* coalescing neighboring slab objects into a single frag which * triggers one of hardened usercopy checks. */ -if (!recv &&page_count (sg_page (sg)) >=1 && !PageSlab (sg_page (sg))) +if (!recv &&sendpage_ok (sg_page (sg)))return ;if (recv) {
Signature · 7 lines
2.26.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v6 6/6 libceph: use sendpage_ok() in ceph_tcp_sendpage()
In reply to Coly LiIn libceph, ceph_tcp_sendpage() does the following checks before handle the page by network layer's zero copy sendpage method,
if (page_count(page) >= 1 && !PageSlab(page))
This check is exactly what sendpage_ok() does. This patch replace the open coded checks by sendpage_ok() as a code cleanup.
--- net/ceph/messenger.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/ceph/messenger.c +1 -1
@@ -575,7 +575,7 @@ static int ceph_tcp_sendpage(struct socket *sock, struct page *page,* coalescing neighboring slab objects into a single frag which * triggers one of hardened usercopy checks. */ -if (page_count (page) >=1 && !PageSlab (page)) +if (sendpage_ok (page)) sendpage = sock->ops->sendpage;else sendpage = sock_no_sendpage;
Signature · 7 lines
2.26.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme