libceph: fix memory leak for messages allocated with CEPH_MSG_DATA_PAGES
PATCH 1/1 libceph: fix memory leak for messages allocated with CEPH_MSG_DATA_PAGES
OSD client allocates a message with a page vector for OSD_MAP, OSD_BACKOFF and WATCH_NOTIFY message types (see alloc_msg_with_page_vector() caller), but pages vector release is never called.
--- net/ceph/messenger.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
net/ceph/messenger.c +8 -1
@@ -3248,8 +3248,15 @@ static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg)static void ceph_msg_data_destroy (struct ceph_msg_data *data) { -if (data->type ==CEPH_MSG_DATA_PAGELIST ) +if (data->type ==CEPH_MSG_DATA_PAGES ) { +int num_pages; + + num_pages =calc_pages_for (data->alignment, + data->length); +ceph_release_page_vector (data->pages, num_pages); + }else if (data->type ==CEPH_MSG_DATA_PAGELIST ) {ceph_pagelist_release (data->pagelist); + } } void ceph_msg_data_add_pages(struct ceph_msg *msg, struct page **pages,
Signature · 1 line
2.24.1
Ilya Dryomov
maintainerQuoted 25 lines · Roman Penyaev
OSD client allocates a message with a page vector for
OSD_MAP,OSD_BACKOFFandWATCH_NOTIFYmessage types (seealloc_msg_with_page_vector()caller), but pages vector release is never called.Signed-off-by Roman Penyaev <[email protected]>Cc Ilya Dryomov <[email protected]>Cc Jeff Layton <[email protected]>Cc Sage Weil <[email protected]>--- net/ceph/messenger.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)net/ceph/messenger.c +2
-1@@ -3248,8 +3248,15 @@ static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg)static void ceph_msg_data_destroy (struct ceph_msg_data *data) { -if (data->type ==CEPH_MSG_DATA_PAGELIST ) +if (data->type ==CEPH_MSG_DATA_PAGES ) { +int num_pages;
net/ceph/messenger.c static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg) · line 3262
Quoted 10 lines
net/ceph/messenger.c +6
-0+ + num_pages =calc_pages_for (data->alignment, + data->length); +ceph_release_page_vector (data->pages, num_pages); + }else if (data->type ==CEPH_MSG_DATA_PAGELIST ) {ceph_pagelist_release (data->pagelist); + } } void ceph_msg_data_add_pages(struct ceph_msg *msg, struct page **pages,
Hi Roman,
I don't think there is a leak here.
osdmap and backoff messages don't have data.
watch_notify message may or may not have data and this is dealt with in handle_watch_notify(). The pages are either released in handle_watch_notify() or transferred to ceph_osdc_notify() through lreq. The caller of ceph_osdc_notify() is then responsible for them:
* @preply_{pages,len} are initialized both on success and error.
* The caller is responsible for:
*
* ceph_release_page_vector(...)
*/
int ceph_osdc_notify(struct ceph_osd_client *osdc,
Thanks,
Ilya
Quoted 36 lines · Ilya Dryomov
Quoted 35 lines · Roman Penyaev
OSD client allocates a message with a page vector for
OSD_MAP,OSD_BACKOFFandWATCH_NOTIFYmessage types (seealloc_msg_with_page_vector()caller), but pages vector release is never called.Signed-off-by Roman Penyaev <[email protected]>Cc Ilya Dryomov <[email protected]>Cc Jeff Layton <[email protected]>Cc Sage Weil <[email protected]>--- net/ceph/messenger.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)net/ceph/messenger.c +8
-1@@ -3248,8 +3248,15 @@ static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg)static void ceph_msg_data_destroy (struct ceph_msg_data *data) { -if (data->type ==CEPH_MSG_DATA_PAGELIST ) +if (data->type ==CEPH_MSG_DATA_PAGES ) { +int num_pages; + + num_pages =calc_pages_for (data->alignment, + data->length); +ceph_release_page_vector (data->pages, num_pages); + }else if (data->type ==CEPH_MSG_DATA_PAGELIST ) {ceph_pagelist_release (data->pagelist); + } }
Quoted 8 lines
Quoted 2 lines
void ceph_msg_data_add_pages(struct
ceph_msg*msg, struct page
**pages,Hi Roman,
I don't think there is a leak here.
osdmap and backoff messages don't have data.
I tried to be symmetric on this path: allocation path exists, but there is no deallocation.
Quoted 6 lines
watch_notifymessage may or may not have data and this is dealt with inhandle_watch_notify(). The pages are either released inhandle_watch_notify()or transferred toceph_osdc_notify()through lreq. The caller ofceph_osdc_notify()is then responsible for them:
Quoted 6 lines
* @preply_{pages,len} are initialized both on success and error.
* The caller is responsible for:* * ceph_release_page_vector(...) */int ceph_osdc_notify(struct
ceph_osd_client*osdc,
Thanks for taking a look. Then there is a rare and unobvious leak, please check ceph_con_in_msg_alloc() in messenger.c. Message can be allocated for osd client (->alloc_msg) and then can be immediately put by the following path:
if (con->state != CON_STATE_OPEN) {
if (msg)
ceph_msg_put(msg);(also few lines below where con->in_msg is put)
without reaching the dispatch and set of handle_* functions, which you've referred.
And seems if the ownership of the ->pages is transferred to the handle_watch_notify() and freed there, then it should be fixed by having release in one place: here or there.
--
Roman
Ilya Dryomov
maintainerQuoted 71 lines · Roman Penyaev
Quoted 39 lines · Ilya Dryomov
Quoted 37 lines · Roman Penyaev
OSD client allocates a message with a page vector for
OSD_MAP,OSD_BACKOFF
andWATCH_NOTIFYmessage types (seealloc_msg_with_page_vector()caller),
but pages vector release is never called.Signed-off-by Roman Penyaev <[email protected]>Cc Ilya Dryomov <[email protected]>Cc Jeff Layton <[email protected]>Cc Sage Weil <[email protected]>--- net/ceph/messenger.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)net/ceph/messenger.c +8
-1@@ -3248,8 +3248,15 @@ static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg)static void ceph_msg_data_destroy (struct ceph_msg_data *data) { -if (data->type ==CEPH_MSG_DATA_PAGELIST ) +if (data->type ==CEPH_MSG_DATA_PAGES ) { +int num_pages; + + num_pages =calc_pages_for (data->alignment, + data->length); +ceph_release_page_vector (data->pages, num_pages); + }else if (data->type ==CEPH_MSG_DATA_PAGELIST ) {ceph_pagelist_release (data->pagelist); + } }Quoted 8 lines
Quoted 2 lines
void ceph_msg_data_add_pages(struct
ceph_msg*msg, struct page
**pages,Hi Roman,
I don't think there is a leak here.
osdmap and backoff messages don't have data.
I tried to be symmetric on this path: allocation path exists, but there is no deallocation.
Quoted 6 lines
watch_notifymessage may or may not have data and this is dealt with inhandle_watch_notify(). The pages are either released inhandle_watch_notify()or transferred toceph_osdc_notify()through lreq. The caller ofceph_osdc_notify()is then responsible for them:Quoted 6 lines
* @preply_{pages,len} are initialized both on success and error.
* The caller is responsible for:* * ceph_release_page_vector(...) */int ceph_osdc_notify(struct
ceph_osd_client*osdc,Thanks for taking a look. Then there is a rare and unobvious leak, please check
ceph_con_in_msg_alloc()inmessenger.c. Message can be allocated for osd client (->alloc_msg) and then can be immediately put by the following path:
Quoted 8 lines
if (con->state != CON_STATE_OPEN) { if (msg) ceph_msg_put(msg);(also few lines below where
con->in_msgis put)without reaching the dispatch and set of handle_* functions, which you've referred.
Yeah, this one is real, and rather old. That is why there is a TODO on top of alloc_msg_with_page_vector():
/*
* TODO: switch to a msg-owned pagelist
*/
Quoted 4 lines
And seems if the ownership of the ->pages is transferred to the
handle_watch_notify()and freed there, then it should be fixed by having release in one place: here or there.
The problem is that at least at one point CEPH_MSG_DATA_PAGES needed a reference count -- it couldn't be freed it in one place. pagelists are reference counted, but can't be easily swapped in, hence that TODO.
Thanks for reminding me about this. I'll take a look -- perhaps the reference count is no longer required and we can get away with a simple flag.
Ilya
On 2020-03-10 11:40, Ilya Dryomov wrote:
[skip]
Quoted 11 lines
Quoted 3 lines
And seems if the ownership of the ->pages is transferred to the
handle_watch_notify()and freed there, then it should be fixed by having release in one place: here or there.The problem is that at least at one point
CEPH_MSG_DATA_PAGESneeded a reference count -- it couldn't be freed it in one place. pagelists are reference counted, but can't be easily swapped in, hence that TODO.Thanks for reminding me about this. I'll take a look -- perhaps the reference count is no longer required and we can get away with a simple flag.
To my shallow understanding handle_watch_notify() also has an error path which eventually does not free or take ownership of data->pages, e.g. callers of 'goto out_unlock_osdc'. Probably code relies on the fact, that sender knows what is doing and never sends ->data with wrong cookie or opcode, but looks very suspicious to me.
Seems for this particular DATA_PAGES case it is possible just to take an ownership by zeroing out data->pages and data->length, which prevents double free, something as the following:
net/ceph/osd_client.c +0 -0
@@ -4440,6 +4440,8 @@ static void handle_watch_notify(struct ceph_osd_client *osdc,
ceph_release_page_vector(data->pages,
calc_pages_for(0,
data->length));
}
+ data->pages = NULL;
+ data->length = 0;
}
lreq->notify_finish_error = return_code;and keeping the current patch with explicit call of ceph_release_page_vector from ceph_msg_data_destroy() from messenger.c.
--
Roman
Ilya Dryomov
maintainerQuoted 44 lines · Roman Penyaev
On 2020-03-10 11:40, Ilya Dryomov wrote:
[skip]
Quoted 11 lines
Quoted 3 lines
And seems if the ownership of the ->pages is transferred to the
handle_watch_notify()and freed there, then it should be fixed by having release in one place: here or there.The problem is that at least at one point
CEPH_MSG_DATA_PAGESneeded a reference count -- it couldn't be freed it in one place. pagelists are reference counted, but can't be easily swapped in, hence that TODO.Thanks for reminding me about this. I'll take a look -- perhaps the reference count is no longer required and we can get away with a simple flag.
To my shallow understanding
handle_watch_notify()also has an error path which eventually does not free or take ownership ofdata->pages, e.g. callers of 'gotoout_unlock_osdc'. Probably code relies on the fact, that sender knows what is doing and never sends ->data with wrong cookie or opcode, but looks very suspicious to me.Seems for this particular
DATA_PAGEScase it is possible just to take an ownership by zeroing outdata->pagesanddata->length, which prevents double free, something as the following:net/ceph/osd_client.c +2
-0@@ -4440,6 +4440,8 @@ static void handle_watch_notify(struct ceph_osd_client *osdc,ceph_release_page_vector (data->pages,calc_pages_for (0 , data->length)); } + data->pages =NULL ; + data->length =0 ;} lreq->notify_finish_error = return_code;
Quoted 3 lines
and keeping the current patch with explicit call of
ceph_release_page_vectorfromceph_msg_data_destroy()frommessenger.c.
Not quite -- that would break all of OSD client, which supplies its own page vectors.
I'll send a patch in a few.
Thanks,
Ilya