Skip to contentSkip to conversation

libceph: fix memory leak for messages allocated with CEPH_MSG_DATA_PAGES

PATCH In review Roman Penyaev 2 participants 6 messages Mar 10, 2020 Ceph common code (libceph)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libceph: fix memory leak for messages allocated with CEPH_MSG_DATA_PAGES

OSD client allocates a message with a page vector for OSD_MAP, OSD_BACKOFF and WATCH_NOTIFY message types (see alloc_msg_with_page_vector() caller), but pages vector release is never called.

---
 net/ceph/messenger.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
net/ceph/messenger.c +8 -1
index 5b4bd8261002..28cbd55ec2e3 100644
@@ -3248,8 +3248,15 @@ static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg)
 
 static void ceph_msg_data_destroy(struct ceph_msg_data *data)
 {
-	if (data->type == CEPH_MSG_DATA_PAGELIST)
+	if (data->type == CEPH_MSG_DATA_PAGES) {
+		int num_pages;
+
+		num_pages = calc_pages_for(data->alignment,
+					   data->length);
+		ceph_release_page_vector(data->pages, num_pages);
+	} else if (data->type == CEPH_MSG_DATA_PAGELIST) {
 		ceph_pagelist_release(data->pagelist);
+	}
 }
 
 void ceph_msg_data_add_pages(struct ceph_msg *msg, struct page **pages,
Signature · 1 line
2.24.1
maintainer
#1 lore raw
Quoted 25 lines · Roman Penyaev

OSD client allocates a message with a page vector for OSD_MAP, OSD_BACKOFF and WATCH_NOTIFY message types (see alloc_msg_with_page_vector() caller), but pages vector release is never called.

---
 net/ceph/messenger.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
net/ceph/messenger.c +2 -1
index 5b4bd8261002..28cbd55ec2e3 100644
@@ -3248,8 +3248,15 @@ static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg)
 
 static void ceph_msg_data_destroy(struct ceph_msg_data *data)
 {
-       if (data->type == CEPH_MSG_DATA_PAGELIST)
+       if (data->type == CEPH_MSG_DATA_PAGES) {
+               int num_pages;

net/ceph/messenger.c static struct ceph_msg_data *ceph_msg_data_add(struct ceph_msg *msg) · line 3262

Quoted 10 lines
net/ceph/messenger.c +6 -0
+
+               num_pages = calc_pages_for(data->alignment,
+                                          data->length);
+               ceph_release_page_vector(data->pages, num_pages);
+       } else if (data->type == CEPH_MSG_DATA_PAGELIST) {
                ceph_pagelist_release(data->pagelist);
+       }
 }
 
 void ceph_msg_data_add_pages(struct ceph_msg *msg, struct page **pages,

Hi Roman,

I don't think there is a leak here.

osdmap and backoff messages don't have data.

watch_notify message may or may not have data and this is dealt with in handle_watch_notify(). The pages are either released in handle_watch_notify() or transferred to ceph_osdc_notify() through lreq. The caller of ceph_osdc_notify() is then responsible for them:

* @preply_{pages,len} are initialized both on success and error.
* The caller is responsible for:
*
* ceph_release_page_vector(...)
*/
int ceph_osdc_notify(struct ceph_osd_client *osdc,

Thanks,

Ilya

#2 lore raw
Quoted 36 lines · Ilya Dryomov
Quoted 35 lines · Roman Penyaev

OSD client allocates a message with a page vector for OSD_MAP, OSD_BACKOFF and WATCH_NOTIFY message types (see alloc_msg_with_page_vector() caller), but pages vector release is never called.

---
 net/ceph/messenger.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
net/ceph/messenger.c +8 -1
index 5b4bd8261002..28cbd55ec2e3 100644
@@ -3248,8 +3248,15 @@ static struct ceph_msg_data 
 *ceph_msg_data_add(struct ceph_msg *msg)
 
 static void ceph_msg_data_destroy(struct ceph_msg_data *data)
 {
-       if (data->type == CEPH_MSG_DATA_PAGELIST)
+       if (data->type == CEPH_MSG_DATA_PAGES) {
+               int num_pages;
+
+               num_pages = calc_pages_for(data->alignment,
+                                          data->length);
+               ceph_release_page_vector(data->pages, num_pages);
+       } else if (data->type == CEPH_MSG_DATA_PAGELIST) {
                ceph_pagelist_release(data->pagelist);
+       }
 }
 
Quoted 8 lines
Quoted 2 lines

void ceph_msg_data_add_pages(struct ceph_msg *msg, struct page
**pages,

Hi Roman,

I don't think there is a leak here.

osdmap and backoff messages don't have data.

I tried to be symmetric on this path: allocation path exists, but there is no deallocation.

Quoted 6 lines

watch_notify message may or may not have data and this is dealt with in handle_watch_notify(). The pages are either released in handle_watch_notify() or transferred to ceph_osdc_notify() through lreq. The caller of ceph_osdc_notify() is then responsible for them:

Quoted 6 lines

* @preply_{pages,len} are initialized both on success and error.
* The caller is responsible for:

   *
   *     ceph_release_page_vector(...)
   */

int ceph_osdc_notify(struct ceph_osd_client *osdc,

Thanks for taking a look. Then there is a rare and unobvious leak, please check ceph_con_in_msg_alloc() in messenger.c. Message can be allocated for osd client (->alloc_msg) and then can be immediately put by the following path:

	if (con->state != CON_STATE_OPEN) {
		if (msg)
			ceph_msg_put(msg);

(also few lines below where con->in_msg is put)

without reaching the dispatch and set of handle_* functions, which you've referred.

And seems if the ownership of the ->pages is transferred to the handle_watch_notify() and freed there, then it should be fixed by having release in one place: here or there.

--
Roman

maintainer
#3 lore raw
Quoted 71 lines · Roman Penyaev
Quoted 39 lines · Ilya Dryomov
Quoted 37 lines · Roman Penyaev

OSD client allocates a message with a page vector for OSD_MAP, OSD_BACKOFF
and WATCH_NOTIFY message types (see alloc_msg_with_page_vector() caller),
but pages vector release is never called.

---
 net/ceph/messenger.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
net/ceph/messenger.c +8 -1
index 5b4bd8261002..28cbd55ec2e3 100644
@@ -3248,8 +3248,15 @@ static struct ceph_msg_data
 *ceph_msg_data_add(struct ceph_msg *msg)
 
 static void ceph_msg_data_destroy(struct ceph_msg_data *data)
 {
-       if (data->type == CEPH_MSG_DATA_PAGELIST)
+       if (data->type == CEPH_MSG_DATA_PAGES) {
+               int num_pages;
+
+               num_pages = calc_pages_for(data->alignment,
+                                          data->length);
+               ceph_release_page_vector(data->pages, num_pages);
+       } else if (data->type == CEPH_MSG_DATA_PAGELIST) {
                ceph_pagelist_release(data->pagelist);
+       }
 }
 
Quoted 8 lines
Quoted 2 lines

void ceph_msg_data_add_pages(struct ceph_msg *msg, struct page
**pages,

Hi Roman,

I don't think there is a leak here.

osdmap and backoff messages don't have data.

I tried to be symmetric on this path: allocation path exists, but there is no deallocation.

Quoted 6 lines

watch_notify message may or may not have data and this is dealt with in handle_watch_notify(). The pages are either released in handle_watch_notify() or transferred to ceph_osdc_notify() through lreq. The caller of ceph_osdc_notify() is then responsible for them:

Quoted 6 lines

* @preply_{pages,len} are initialized both on success and error.
* The caller is responsible for:

   *
   *     ceph_release_page_vector(...)
   */

int ceph_osdc_notify(struct ceph_osd_client *osdc,

Thanks for taking a look. Then there is a rare and unobvious leak, please check ceph_con_in_msg_alloc() in messenger.c. Message can be allocated for osd client (->alloc_msg) and then can be immediately put by the following path:

Quoted 8 lines
        if (con->state != CON_STATE_OPEN) {
                if (msg)
                        ceph_msg_put(msg);

(also few lines below where con->in_msg is put)

without reaching the dispatch and set of handle_* functions, which you've referred.

Yeah, this one is real, and rather old. That is why there is a TODO on top of alloc_msg_with_page_vector():

/*
* TODO: switch to a msg-owned pagelist
*/

Quoted 4 lines

And seems if the ownership of the ->pages is transferred to the handle_watch_notify() and freed there, then it should be fixed by having release in one place: here or there.

The problem is that at least at one point CEPH_MSG_DATA_PAGES needed a reference count -- it couldn't be freed it in one place. pagelists are reference counted, but can't be easily swapped in, hence that TODO.

Thanks for reminding me about this. I'll take a look -- perhaps the reference count is no longer required and we can get away with a simple flag.

Ilya

#4 lore raw

On 2020-03-10 11:40, Ilya Dryomov wrote:

[skip]

Quoted 11 lines
Quoted 3 lines

And seems if the ownership of the ->pages is transferred to the handle_watch_notify() and freed there, then it should be fixed by having release in one place: here or there.

The problem is that at least at one point CEPH_MSG_DATA_PAGES needed a reference count -- it couldn't be freed it in one place. pagelists are reference counted, but can't be easily swapped in, hence that TODO.

Thanks for reminding me about this. I'll take a look -- perhaps the reference count is no longer required and we can get away with a simple flag.

To my shallow understanding handle_watch_notify() also has an error path which eventually does not free or take ownership of data->pages, e.g. callers of 'goto out_unlock_osdc'. Probably code relies on the fact, that sender knows what is doing and never sends ->data with wrong cookie or opcode, but looks very suspicious to me.

Seems for this particular DATA_PAGES case it is possible just to take an ownership by zeroing out data->pages and data->length, which prevents double free, something as the following:

net/ceph/osd_client.c +0 -0
index b68b376d8c2f..15ae6377c461 100644
@@ -4440,6 +4440,8 @@ static void handle_watch_notify(struct ceph_osd_client *osdc,
                                        

ceph_release_page_vector(data->pages,

                                               calc_pages_for(0, 
data->length));
                                }
+                               data->pages = NULL;
+                               data->length = 0;
                        }
                        lreq->notify_finish_error = return_code;

and keeping the current patch with explicit call of ceph_release_page_vector from ceph_msg_data_destroy() from messenger.c.

--
Roman

maintainer
#5 lore raw
Quoted 44 lines · Roman Penyaev

On 2020-03-10 11:40, Ilya Dryomov wrote:

[skip]

Quoted 11 lines
Quoted 3 lines

And seems if the ownership of the ->pages is transferred to the handle_watch_notify() and freed there, then it should be fixed by having release in one place: here or there.

The problem is that at least at one point CEPH_MSG_DATA_PAGES needed a reference count -- it couldn't be freed it in one place. pagelists are reference counted, but can't be easily swapped in, hence that TODO.

Thanks for reminding me about this. I'll take a look -- perhaps the reference count is no longer required and we can get away with a simple flag.

To my shallow understanding handle_watch_notify() also has an error path which eventually does not free or take ownership of data->pages, e.g. callers of 'goto out_unlock_osdc'. Probably code relies on the fact, that sender knows what is doing and never sends ->data with wrong cookie or opcode, but looks very suspicious to me.

Seems for this particular DATA_PAGES case it is possible just to take an ownership by zeroing out data->pages and data->length, which prevents double free, something as the following:

net/ceph/osd_client.c +2 -0
index b68b376d8c2f..15ae6377c461 100644
@@ -4440,6 +4440,8 @@ static void handle_watch_notify(struct
 ceph_osd_client *osdc,
 
 ceph_release_page_vector(data->pages,
                                                calc_pages_for(0,
 data->length));
                                 }
+                               data->pages = NULL;
+                               data->length = 0;
                         }
                         lreq->notify_finish_error = return_code;
Quoted 3 lines

and keeping the current patch with explicit call of ceph_release_page_vector from ceph_msg_data_destroy() from messenger.c.

Not quite -- that would break all of OSD client, which supplies its own page vectors.

I'll send a patch in a few.

Thanks,

Ilya