Skip to contentSkip to conversation

libceph: per-op osd request data

PATCH In review Alex Elder 2 participants 41 messages Apr 5, 2013 – Apr 8, 2013

Subsystems (4)
lore
Share & tools
lore

Alex Elder

author
#0 lore raw

cover libceph: per-op osd request data

This series of patches arranges for each op in an osd request to define its own incoming and outgoing data. Doing so allows a single request to contain muliple ops which provide outgoing data, and even allows a single op to supply more than one item of data.

A bunch of the patches here are fairly trivial cleanups, done as I was figuring out how to get things done. But in the end the changes involved are significant so I tried to do them in methodical steps.

These patches (and the one I just posted) are available in the branch "review/wip-3761-1" in the ceph-client git repository.

-Alex

[PATCH 01/20] rbd: define inbound data size for method ops
Whoops, this one's not directly related to the rest, I intended to post this one by itself.

[PATCH 02/20] libceph: compute incoming bytes once [PATCH 03/20] libceph: define osd data initialization helpers [PATCH 04/20] libceph: define a few more helpers [PATCH 05/20] libceph: define ceph_osd_data_length() [PATCH 06/20] libceph: a few more osd data cleanups
Simple cleanups, and code encapsulation.

[PATCH 07/20] rbd: define rbd_osd_req_format_op()
[PATCH 08/20] libceph: keep source rather than message osd op array
Store the source op array in the osd request structure.

[PATCH 09/20] libceph: rename data out field in osd request op [PATCH 10/20] libceph: add data pointers in osd op structures [PATCH 11/20] libceph: specify osd op by index in request
Start defining op data in the op structure itself.

[PATCH 12/20] rbd: don't set data in rbd_osd_req_format_op() [PATCH 13/20] rbd: separate initialization of osd data [PATCH 14/20] rbd: rearrange some code for consistency
Changes to rbd so it doesn't assume single op per request.

[PATCH 15/20] libceph: format class info at init time
Don't wait until formatting the op to lay out the request information (class, method, and data). Define a new data item in the CALL op structure to represent this.

[PATCH 16/20] libceph: move ceph_osdc_build_request() [PATCH 17/20] libceph: set message data when building osd request [PATCH 18/20] libceph: combine initializing and setting osd data [PATCH 19/20] libceph: set the data pointers when encoding ops [PATCH 20/20] libceph: kill off osd request r_data_in and r_data_out
These last five (four really) finally make the switch over to using data item descriptions present in the ops to dictate what's used in osd request and response messages.

author
#1 lore raw

PATCH 1/20 rbd: define inbound data size for method ops

When rbd creates an object request containing an object method call operation it is passing 0 for the size. I originally thought this was because the length was not needed for method calls, but I think it really should be supplied, to describe how much space is available to receive response data. So provide the supplied length.

This resolves:

    http://tracker.ceph.com/issues/4659
---
 drivers/block/rbd.c |   13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/block/rbd.c +0 -0
index 9fb51b5..5e579fa 100644
@@ -1839,12 +1839,11 @@ static int rbd_obj_method_sync(struct rbd_device

*rbd_dev,
int ret;

/*
- * Method calls are ultimately read operations but they
- * don't involve object data (so no offset or length).
- * The result should placed into the inbound buffer
- * provided. They also supply outbound data--parameters for
- * the object method. Currently if this is present it will
- * be a snapshot id.
+ * Method calls are ultimately read operations. The result
+ * should placed into the inbound buffer provided. They
+ * also supply outbound data--parameters for the object
+ * method. Currently if this is present it will be a
+ * snapshot id.
*/
page_count = (u32) calc_pages_for(0, inbound_size); pages = ceph_alloc_page_vector(page_count, GFP_KERNEL);
@@ -1852,7 +1851,7 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
return PTR_ERR(pages);

ret = -ENOMEM;
- obj_request = rbd_obj_request_create(object_name, 0, 0,
+ obj_request = rbd_obj_request_create(object_name, 0, inbound_size,
OBJ_REQUEST_PAGES);
if (!obj_request)
goto out;

Signature · 1 line
1.7.9.5
Reviewed-by on v1 1/20
#2 lore raw

Re: [PATCH 01/20] rbd: define inbound data size for method ops

Quoted 48 lines · Alex Elder

When rbd creates an object request containing an object method call operation it is passing 0 for the size. I originally thought this was because the length was not needed for method calls, but I think it really should be supplied, to describe how much space is available to receive response data. So provide the supplied length.

This resolves:

     http://tracker.ceph.com/issues/4659
---
  drivers/block/rbd.c |   13 ++++++-------
  1 file changed, 6 insertions(+), 7 deletions(-)
drivers/block/rbd.c +5 -6
index 9fb51b5..5e579fa 100644
@@ -1839,12 +1839,11 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  	int ret;
 
  	/*
-	 * Method calls are ultimately read operations but they
-	 * don't involve object data (so no offset or length).
-	 * The result should placed into the inbound buffer
-	 * provided.  They also supply outbound data--parameters for
-	 * the object method.  Currently if this is present it will
-	 * be a snapshot id.
+	 * Method calls are ultimately read operations.  The result
+	 * should placed into the inbound buffer provided.  They
+	 * also supply outbound data--parameters for the object
+	 * method.  Currently if this is present it will be a
+	 * snapshot id.
  	 */
  	page_count = (u32) calc_pages_for(0, inbound_size);
  	pages = ceph_alloc_page_vector(page_count, GFP_KERNEL);
drivers/block/rbd.c +1 -1
@@ -1852,7 +1851,7 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  		return PTR_ERR(pages);
 
  	ret = -ENOMEM;
-	obj_request = rbd_obj_request_create(object_name, 0, 0,
+	obj_request = rbd_obj_request_create(object_name, 0, inbound_size,
  							OBJ_REQUEST_PAGES);
  	if (!obj_request)

goto out;

author
#3 lore raw

PATCH 2/20 libceph: compute incoming bytes once

In reply to Alex Elder

This is a simple change, extracting the number of incoming data bytes just once in handle_reply().

---
 net/ceph/osd_client.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
net/ceph/osd_client.c +0 -0
index 426ca1f..1379b33 100644
@@ -1293,6 +1293,7 @@ static void handle_reply(struct ceph_osd_client

*osdc, struct ceph_msg *msg,
u64 reassert_version;
u32 osdmap_epoch;
int already_completed;
+ u32 bytes;
int i;

tid = le64_to_cpu(msg->hdr.tid);
@@ -1347,9 +1348,10 @@ static void handle_reply(struct ceph_osd_client *osdc, struct ceph_msg *msg,
payload_len += len;
p += sizeof(*op);
}
- if (payload_len != le32_to_cpu(msg->hdr.data_len)) {
+ bytes = le32_to_cpu(msg->hdr.data_len);
+ if (payload_len != bytes) {
pr_warning("sum of op payload lens %d != data_len %d",
- payload_len, le32_to_cpu(msg->hdr.data_len));
+ payload_len, bytes);
goto bad_put;
}

@@ -1359,10 +1361,8 @@ static void handle_reply(struct ceph_osd_client *osdc, struct ceph_msg *msg,
req->r_reply_op_result[i] = ceph_decode_32(&p);

if (!req->r_got_reply) {
- unsigned int bytes;

req->r_result = result;
- bytes = le32_to_cpu(msg->hdr.data_len);
dout("handle_reply result %d bytes %d\n", req->r_result,
bytes);
if (req->r_result == 0)

Signature · 1 line
1.7.9.5
Reviewed-by on v1 2/20
#4 lore raw

Re: [PATCH 02/20] libceph: compute incoming bytes once

Quoted 48 lines · Alex Elder

This is a simple change, extracting the number of incoming data bytes just once in handle_reply().

---
  net/ceph/osd_client.c |    8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
net/ceph/osd_client.c +1 -0
index 426ca1f..1379b33 100644
@@ -1293,6 +1293,7 @@ static void handle_reply(struct ceph_osd_client
 *osdc, struct ceph_msg *msg,
  	u64 reassert_version;
  	u32 osdmap_epoch;
  	int already_completed;
+	u32 bytes;
  	int i;
 
  	tid = le64_to_cpu(msg->hdr.tid);
net/ceph/osd_client.c +3 -4
@@ -1347,9 +1348,10 @@ static void handle_reply(struct ceph_osd_client
 *osdc, struct ceph_msg *msg,
  		payload_len += len;
  		p += sizeof(*op);
  	}
-	if (payload_len != le32_to_cpu(msg->hdr.data_len)) {
+	bytes = le32_to_cpu(msg->hdr.data_len);
+	if (payload_len != bytes) {
  		pr_warning("sum of op payload lens %d != data_len %d",
-			   payload_len, le32_to_cpu(msg->hdr.data_len));
+			   payload_len, bytes);
  		goto bad_put;
  	}
@@ -1359,10 +1361,8 @@ static void handle_reply(struct ceph_osd_client
 *osdc, struct ceph_msg *msg,
  		req->r_reply_op_result[i] = ceph_decode_32(&p);
 
  	if (!req->r_got_reply) {
-		unsigned int bytes;
 
  		req->r_result = result;
-		bytes = le32_to_cpu(msg->hdr.data_len);
  		dout("handle_reply result %d bytes %d\n", req->r_result,
  		     bytes);

if (req->r_result == 0)

author
#5 lore raw

PATCH 3/20 libceph: define osd data initialization helpers

In reply to Alex Elder

Define and use functions that encapsulate the initializion of a ceph_osd_data structure.

---
 drivers/block/rbd.c             |   14 ++++------
 fs/ceph/addr.c                  |   13 +++------
 fs/ceph/file.c                  |   10 +++----
 include/linux/ceph/osd_client.h |   11 ++++++++
 net/ceph/osd_client.c           |   55
+++++++++++++++++++++++++++------------
 5 files changed, 63 insertions(+), 40 deletions(-)
drivers/block/rbd.c +5 -9
index 5e579fa..a9d88a0 100644
@@ -1350,17 +1350,13 @@ static struct ceph_osd_request *rbd_osd_req_create(
 		break;		/* Nothing to do */
 	case OBJ_REQUEST_BIO:
 		rbd_assert(obj_request->bio_list != NULL);
-		osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
-		osd_data->bio = obj_request->bio_list;
-		osd_data->bio_length = obj_request->length;
+		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
+					obj_request->length);
 		break;
 	case OBJ_REQUEST_PAGES:
-		osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
-		osd_data->pages = obj_request->pages;
-		osd_data->length = obj_request->length;
-		osd_data->alignment = offset & ~PAGE_MASK;
-		osd_data->pages_from_pool = false;
-		osd_data->own_pages = false;
+		ceph_osd_data_pages_init(osd_data, obj_request->pages,
+				obj_request->length, offset & ~PAGE_MASK,
+				false, false);
 		break;
 	}
 
fs/ceph/addr.c +0 -0
index e0dd74c..8507389 100644
@@ -342,10 +342,8 @@ static int start_read(struct inode *inode, struct

list_head *page_list, int max)
}
pages[i] = page;
}
- req->r_data_in.type = CEPH_OSD_DATA_TYPE_PAGES;
- req->r_data_in.pages = pages;
- req->r_data_in.length = len;
- req->r_data_in.alignment = 0;
+ ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
+ false, false);
req->r_callback = finish_read;
req->r_inode = inode;

@@ -917,11 +915,8 @@ get_more_pages:
dout("writepages got %d pages at %llu~%llu\n",
locked_pages, offset, len);

- req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
- req->r_data_out.pages = pages;
- req->r_data_out.length = len;
- req->r_data_out.alignment = 0;
- req->r_data_out.pages_from_pool = !!pool;
+ ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
+ !!pool, false);

pages = NULL; /* request message now owns the pages array */
pool = NULL;

fs/ceph/file.c +0 -0
index 66b8469..2f2d0a1 100644
@@ -491,6 +491,7 @@ static ssize_t ceph_sync_write(struct file *file,

const char __user *data,
unsigned long buf_align;
int ret;
struct timespec mtime = CURRENT_TIME;
+ bool own_pages = false;

if (ceph_snap(file->f_dentry->d_inode) != CEPH_NOSNAP)
return -EROFS;
@@ -571,14 +572,11 @@ more:
if ((file->f_flags & O_SYNC) == 0) {
/* get a second commit callback */ req->r_safe_callback = sync_write_commit;
- req->r_data_out.own_pages = 1;
+ own_pages = true;
}
}
- req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
- req->r_data_out.pages = pages;
- req->r_data_out.length = len;
- req->r_data_out.alignment = page_align;
- req->r_inode = inode;
+ ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+ false, own_pages);

/* BUG_ON(vino.snap != CEPH_NOSNAP); */ ceph_osdc_build_request(req, pos, num_ops, ops,

(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index 5ee1a37..af60dac 100644

include/linux/ceph/osd_client.h +0 -0
@@ -280,6 +280,17 @@ static inline void ceph_osdc_put_request(struct

ceph_osd_request *req)
kref_put(&req->r_kref, ceph_osdc_release_request);
}

+extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,

+				     struct page **pages, u64 length,
+				     u32 alignment, bool pages_from_pool,
+				     bool own_pages);

+extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+ struct ceph_pagelist *pagelist);
+#ifdef CONFIG_BLOCK
+extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+ struct bio *bio, size_t bio_length);
+#endif /* CONFIG_BLOCK */
+
extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
struct ceph_osd_request *req, bool nofail);

net/ceph/osd_client.c +0 -0
index 1379b33..f8f8561 100644
@@ -79,6 +79,38 @@ static int calc_layout(struct ceph_file_layout

*layout, u64 off, u64 *plen,
return 0;
}

+void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
+ struct page **pages, u64 length, u32 alignment,
+ bool pages_from_pool, bool own_pages)
+{
+ osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
+ osd_data->pages = pages;
+ osd_data->length = length;
+ osd_data->alignment = alignment;
+ osd_data->pages_from_pool = pages_from_pool;
+ osd_data->own_pages = own_pages;
+}
+EXPORT_SYMBOL(ceph_osd_data_pages_init);
+
+void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+ struct ceph_pagelist *pagelist)
+{
+ osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST;
+ osd_data->pagelist = pagelist;
+}
+EXPORT_SYMBOL(ceph_osd_data_pagelist_init);
+
+#ifdef CONFIG_BLOCK
+void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+ struct bio *bio, size_t bio_length)
+{
+ osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
+ osd_data->bio = bio;
+ osd_data->bio_length = bio_length;
+}
+EXPORT_SYMBOL(ceph_osd_data_bio_init);
+#endif /* CONFIG_BLOCK */
+
/*
* requests
*/
@@ -400,8 +432,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
ceph_pagelist_append(pagelist, src->cls.indata,
src->cls.indata_len);

- req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGELIST;
- req->r_data_out.pagelist = pagelist;
+ ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
out_data_len = pagelist->length;
break;
case CEPH_OSD_OP_STARTSYNC:
@@ -2056,7 +2087,6 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
struct page **pages, int num_pages, int page_align)
{
struct ceph_osd_request *req;
- struct ceph_osd_data *osd_data;
struct ceph_osd_req_op op;
int rc = 0;

@@ -2071,14 +2101,11 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,

/* it may be a short read due to an object boundary */

- osd_data = &req->r_data_in;
- osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
- osd_data->pages = pages;
- osd_data->length = *plen;
- osd_data->alignment = page_align;
+ ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
+ false, false);

dout("readpages final extent is %llu~%llu (%llu bytes align %d)\n",
- off, *plen, osd_data->length, page_align);
+ off, *plen, *plen, page_align);

ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);

@@ -2104,7 +2131,6 @@ int ceph_osdc_writepages(struct ceph_osd_client *osdc, struct ceph_vino vino,
struct page **pages, int num_pages)
{
struct ceph_osd_request *req;
- struct ceph_osd_data *osd_data;
struct ceph_osd_req_op op;
int rc = 0;
int page_align = off & ~PAGE_MASK;
@@ -2119,12 +2145,9 @@ int ceph_osdc_writepages(struct ceph_osd_client *osdc, struct ceph_vino vino,
return PTR_ERR(req);

/* it may be a short write due to an object boundary */
- osd_data = &req->r_data_out;
- osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
- osd_data->pages = pages;
- osd_data->length = len;
- osd_data->alignment = page_align;
- dout("writepages %llu~%llu (%llu bytes)\n", off, len, osd_data->length);
+ ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+ false, false);
+ dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);

ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);

Signature · 1 line
1.7.9.5
Reviewed-by on v1 3/20
#6 lore raw

Re: [PATCH 03/20] libceph: define osd data initialization helpers

Quoted 235 lines · Alex Elder

Define and use functions that encapsulate the initializion of a ceph_osd_data structure.

---
  drivers/block/rbd.c             |   14 ++++------
  fs/ceph/addr.c                  |   13 +++------
  fs/ceph/file.c                  |   10 +++----
  include/linux/ceph/osd_client.h |   11 ++++++++
  net/ceph/osd_client.c           |   55
+++++++++++++++++++++++++++------------
  5 files changed, 63 insertions(+), 40 deletions(-)
drivers/block/rbd.c +5 -9
index 5e579fa..a9d88a0 100644
@@ -1350,17 +1350,13 @@ static struct ceph_osd_request *rbd_osd_req_create(
  		break;		/* Nothing to do */
  	case OBJ_REQUEST_BIO:
  		rbd_assert(obj_request->bio_list != NULL);
-		osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
-		osd_data->bio = obj_request->bio_list;
-		osd_data->bio_length = obj_request->length;
+		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
+					obj_request->length);
  		break;
  	case OBJ_REQUEST_PAGES:
-		osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
-		osd_data->pages = obj_request->pages;
-		osd_data->length = obj_request->length;
-		osd_data->alignment = offset & ~PAGE_MASK;
-		osd_data->pages_from_pool = false;
-		osd_data->own_pages = false;
+		ceph_osd_data_pages_init(osd_data, obj_request->pages,
+				obj_request->length, offset & ~PAGE_MASK,
+				false, false);
  		break;
  	}
 
fs/ceph/addr.c +4 -9
index e0dd74c..8507389 100644
@@ -342,10 +342,8 @@ static int start_read(struct inode *inode, struct
 list_head *page_list, int max)
  		}
  		pages[i] = page;
  	}
-	req->r_data_in.type = CEPH_OSD_DATA_TYPE_PAGES;
-	req->r_data_in.pages = pages;
-	req->r_data_in.length = len;
-	req->r_data_in.alignment = 0;
+	ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
+					false, false);
  	req->r_callback = finish_read;
  	req->r_inode = inode;
@@ -917,11 +915,8 @@ get_more_pages:
  		dout("writepages got %d pages at %llu~%llu\n",
  		     locked_pages, offset, len);
 
-		req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
-		req->r_data_out.pages = pages;
-		req->r_data_out.length = len;
-		req->r_data_out.alignment = 0;
-		req->r_data_out.pages_from_pool = !!pool;
+		ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
+						!!pool, false);
 
  		pages = NULL;	/* request message now owns the pages array */
  		pool = NULL;
fs/ceph/file.c +1 -0
index 66b8469..2f2d0a1 100644
@@ -491,6 +491,7 @@ static ssize_t ceph_sync_write(struct file *file,
 const char __user *data,
  	unsigned long buf_align;
  	int ret;
  	struct timespec mtime = CURRENT_TIME;
+	bool own_pages = false;
 
  	if (ceph_snap(file->f_dentry->d_inode) != CEPH_NOSNAP)

return -EROFS;

fs/ceph/file.c +3 -6
@@ -571,14 +572,11 @@ more:
  		if ((file->f_flags & O_SYNC) == 0) {
  			/* get a second commit callback */
  			req->r_safe_callback = sync_write_commit;
-			req->r_data_out.own_pages = 1;
+			own_pages = true;
  		}
  	}
-	req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGES;
-	req->r_data_out.pages = pages;
-	req->r_data_out.length = len;
-	req->r_data_out.alignment = page_align;
-	req->r_inode = inode;
+	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+					false, own_pages);
 
  	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
  	ceph_osdc_build_request(req, pos, num_ops, ops,
include/linux/ceph/osd_client.h +11 -0
index 5ee1a37..af60dac 100644
@@ -280,6 +280,17 @@ static inline void ceph_osdc_put_request(struct
 ceph_osd_request *req)
  	kref_put(&req->r_kref, ceph_osdc_release_request);
  }
 
+extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
+				     struct page **pages, u64 length,
+				     u32 alignment, bool pages_from_pool,
+				     bool own_pages);
+extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+					struct ceph_pagelist *pagelist);
+#ifdef CONFIG_BLOCK
+extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+				   struct bio *bio, size_t bio_length);
+#endif /* CONFIG_BLOCK */
+
  extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
  				   struct ceph_osd_request *req,

bool nofail);

net/ceph/osd_client.c +32 -0
index 1379b33..f8f8561 100644
@@ -79,6 +79,38 @@ static int calc_layout(struct ceph_file_layout
 *layout, u64 off, u64 *plen,
  	return 0;
  }
 
+void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
+			struct page **pages, u64 length, u32 alignment,
+			bool pages_from_pool, bool own_pages)
+{
+	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
+	osd_data->pages = pages;
+	osd_data->length = length;
+	osd_data->alignment = alignment;
+	osd_data->pages_from_pool = pages_from_pool;
+	osd_data->own_pages = own_pages;
+}
+EXPORT_SYMBOL(ceph_osd_data_pages_init);
+
+void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+			struct ceph_pagelist *pagelist)
+{
+	osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST;
+	osd_data->pagelist = pagelist;
+}
+EXPORT_SYMBOL(ceph_osd_data_pagelist_init);
+
+#ifdef CONFIG_BLOCK
+void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+			struct bio *bio, size_t bio_length)
+{
+	osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
+	osd_data->bio = bio;
+	osd_data->bio_length = bio_length;
+}
+EXPORT_SYMBOL(ceph_osd_data_bio_init);
+#endif /* CONFIG_BLOCK */
+
  /*
   * requests
   */
net/ceph/osd_client.c +1 -2
@@ -400,8 +432,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
 *req,
  		ceph_pagelist_append(pagelist, src->cls.indata,
  				     src->cls.indata_len);
 
-		req->r_data_out.type = CEPH_OSD_DATA_TYPE_PAGELIST;
-		req->r_data_out.pagelist = pagelist;
+		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
  		out_data_len = pagelist->length;
  		break;

case CEPH_OSD_OP_STARTSYNC:

net/ceph/osd_client.c +3 -8
@@ -2056,7 +2087,6 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
  			struct page **pages, int num_pages, int page_align)
  {
  	struct ceph_osd_request *req;
-	struct ceph_osd_data *osd_data;
  	struct ceph_osd_req_op op;
  	int rc = 0;
 
@@ -2071,14 +2101,11 @@ int ceph_osdc_readpages(struct ceph_osd_client
 *osdc,
 
  	/* it may be a short read due to an object boundary */
 
-	osd_data = &req->r_data_in;
-	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
-	osd_data->pages = pages;
-	osd_data->length = *plen;
-	osd_data->alignment = page_align;
+	ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
+				false, false);
 
  	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
-	     off, *plen, osd_data->length, page_align);
+	     off, *plen, *plen, page_align);
 
  	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
@@ -2104,7 +2131,6 @@ int ceph_osdc_writepages(struct ceph_osd_client
 *osdc, struct ceph_vino vino,
  			 struct page **pages, int num_pages)
  {
  	struct ceph_osd_request *req;
-	struct ceph_osd_data *osd_data;
  	struct ceph_osd_req_op op;
  	int rc = 0;
  	int page_align = off & ~PAGE_MASK;
net/ceph/osd_client.c +3 -6
@@ -2119,12 +2145,9 @@ int ceph_osdc_writepages(struct ceph_osd_client
 *osdc, struct ceph_vino vino,
  		return PTR_ERR(req);
 
  	/* it may be a short write due to an object boundary */
-	osd_data = &req->r_data_out;
-	osd_data->type = CEPH_OSD_DATA_TYPE_PAGES;
-	osd_data->pages = pages;
-	osd_data->length = len;
-	osd_data->alignment = page_align;
-	dout("writepages %llu~%llu (%llu bytes)\n", off, len, osd_data->length);
+	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+				false, false);
+	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);
 
  	ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);
author
#7 lore raw

PATCH 4/20 libceph: define a few more helpers

In reply to Alex Elder

Define ceph_osd_data_init() and ceph_osd_data_release() to clean up a little code.

---
 net/ceph/osd_client.c |   44 ++++++++++++++++++++++++++------------------
 1 file changed, 26 insertions(+), 18 deletions(-)
net/ceph/osd_client.c +0 -0
index f8f8561..df0f856 100644
@@ -79,6 +79,12 @@ static int calc_layout(struct ceph_file_layout

*layout, u64 off, u64 *plen,
return 0;
}

+static void ceph_osd_data_init(struct ceph_osd_data *osd_data) +{
+ memset(osd_data, 0, sizeof *osd_data);
+ osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
+}
+
void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
struct page **pages, u64 length, u32 alignment, bool pages_from_pool, bool own_pages)
@@ -111,16 +117,28 @@ void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
EXPORT_SYMBOL(ceph_osd_data_bio_init);
#endif /* CONFIG_BLOCK */

+static void ceph_osd_data_release(struct ceph_osd_data *osd_data) +{
+ if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
+ return;
+
+ if (osd_data->own_pages) {
+ int num_pages;
+
+ num_pages = calc_pages_for((u64)osd_data->alignment,
+ (u64)osd_data->length);
+ ceph_release_page_vector(osd_data->pages, num_pages);
+ }
+}
+
/*
* requests
*/
void ceph_osdc_release_request(struct kref *kref)
{
- int num_pages;
- struct ceph_osd_request *req = container_of(kref,
- struct ceph_osd_request,
- r_kref);
+ struct ceph_osd_request *req;

+ req = container_of(kref, struct ceph_osd_request, r_kref);
if (req->r_request)
ceph_msg_put(req->r_request);
if (req->r_reply) {
@@ -128,18 +146,8 @@ void ceph_osdc_release_request(struct kref *kref)
ceph_msg_put(req->r_reply);
}

- if (req->r_data_in.type == CEPH_OSD_DATA_TYPE_PAGES &&
- req->r_data_in.own_pages) {
- num_pages = calc_pages_for((u64)req->r_data_in.alignment,
- (u64)req->r_data_in.length);
- ceph_release_page_vector(req->r_data_in.pages, num_pages);
- }
- if (req->r_data_out.type == CEPH_OSD_DATA_TYPE_PAGES &&
- req->r_data_out.own_pages) {
- num_pages = calc_pages_for((u64)req->r_data_out.alignment,
- (u64)req->r_data_out.length);
- ceph_release_page_vector(req->r_data_out.pages, num_pages);
- }
+ ceph_osd_data_release(&req->r_data_in);
+ ceph_osd_data_release(&req->r_data_out);

ceph_put_snap_context(req->r_snapc); if (req->r_mempool)
@@ -203,8 +211,8 @@ struct ceph_osd_request *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
}
req->r_reply = msg;

- req->r_data_in.type = CEPH_OSD_DATA_TYPE_NONE;
- req->r_data_out.type = CEPH_OSD_DATA_TYPE_NONE;
+ ceph_osd_data_init(&req->r_data_in);
+ ceph_osd_data_init(&req->r_data_out);

/* create request message; allow space for oid */ if (use_mempool)

Signature · 1 line
1.7.9.5
Reviewed-by on v1 4/20
#8 lore raw

Re: [PATCH 04/20] libceph: define a few more helpers

There's a sizeof without parentheses, but after fixing that:

Quoted 94 lines · Alex Elder

Define ceph_osd_data_init() and ceph_osd_data_release() to clean up a little code.

---
  net/ceph/osd_client.c |   44 ++++++++++++++++++++++++++------------------
  1 file changed, 26 insertions(+), 18 deletions(-)
net/ceph/osd_client.c +6 -0
index f8f8561..df0f856 100644
@@ -79,6 +79,12 @@ static int calc_layout(struct ceph_file_layout
 *layout, u64 off, u64 *plen,
  	return 0;
  }
 
+static void ceph_osd_data_init(struct ceph_osd_data *osd_data)
+{
+	memset(osd_data, 0, sizeof *osd_data);
+	osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
+}
+
  void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
  			struct page **pages, u64 length, u32 alignment,

bool pages_from_pool, bool own_pages)

net/ceph/osd_client.c +16 -4
@@ -111,16 +117,28 @@ void ceph_osd_data_bio_init(struct ceph_osd_data
 *osd_data,
  EXPORT_SYMBOL(ceph_osd_data_bio_init);
  #endif /* CONFIG_BLOCK */
 
+static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
+{
+	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
+		return;
+
+	if (osd_data->own_pages) {
+		int num_pages;
+
+		num_pages = calc_pages_for((u64)osd_data->alignment,
+						(u64)osd_data->length);
+		ceph_release_page_vector(osd_data->pages, num_pages);
+	}
+}
+
  /*
   * requests
   */
  void ceph_osdc_release_request(struct kref *kref)
  {
-	int num_pages;
-	struct ceph_osd_request *req = container_of(kref,
-						    struct ceph_osd_request,
-						    r_kref);
+	struct ceph_osd_request *req;
 
+	req = container_of(kref, struct ceph_osd_request, r_kref);
  	if (req->r_request)
  		ceph_msg_put(req->r_request);
  	if (req->r_reply) {
net/ceph/osd_client.c +4 -14
@@ -128,18 +146,8 @@ void ceph_osdc_release_request(struct kref *kref)
  		ceph_msg_put(req->r_reply);
  	}
 
-	if (req->r_data_in.type == CEPH_OSD_DATA_TYPE_PAGES &&
-			req->r_data_in.own_pages) {
-		num_pages = calc_pages_for((u64)req->r_data_in.alignment,
-						(u64)req->r_data_in.length);
-		ceph_release_page_vector(req->r_data_in.pages, num_pages);
-	}
-	if (req->r_data_out.type == CEPH_OSD_DATA_TYPE_PAGES &&
-			req->r_data_out.own_pages) {
-		num_pages = calc_pages_for((u64)req->r_data_out.alignment,
-						(u64)req->r_data_out.length);
-		ceph_release_page_vector(req->r_data_out.pages, num_pages);
-	}
+	ceph_osd_data_release(&req->r_data_in);
+	ceph_osd_data_release(&req->r_data_out);
 
  	ceph_put_snap_context(req->r_snapc);
  	if (req->r_mempool)
@@ -203,8 +211,8 @@ struct ceph_osd_request
 *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
  	}
  	req->r_reply = msg;
 
-	req->r_data_in.type = CEPH_OSD_DATA_TYPE_NONE;
-	req->r_data_out.type = CEPH_OSD_DATA_TYPE_NONE;
+	ceph_osd_data_init(&req->r_data_in);
+	ceph_osd_data_init(&req->r_data_out);
 
  	/* create request message; allow space for oid */

if (use_mempool)

author
#9 lore raw

PATCH 5/20 libceph: define ceph_osd_data_length()

In reply to Alex Elder

One more osd data helper, which returns the length of the data item, regardless of its type.

---
 net/ceph/osd_client.c |   31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)
net/ceph/osd_client.c +0 -0
index df0f856..110cd83 100644
@@ -117,6 +117,25 @@ void ceph_osd_data_bio_init(struct ceph_osd_data

*osd_data,
EXPORT_SYMBOL(ceph_osd_data_bio_init);
#endif /* CONFIG_BLOCK */

+static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data) +{
+ switch (osd_data->type) {
+ case CEPH_OSD_DATA_TYPE_NONE:
+ return 0;
+ case CEPH_OSD_DATA_TYPE_PAGES:
+ return osd_data->length;
+ case CEPH_OSD_DATA_TYPE_PAGELIST:
+ return (u64)osd_data->pagelist->length;
+#ifdef CONFIG_BLOCK
+ case CEPH_OSD_DATA_TYPE_BIO:
+ return (u64)osd_data->bio_length;
+#endif /* CONFIG_BLOCK */
+ default:
+ WARN(true, "unrecognized data type %d\n", (int)osd_data->type);
+ return 0;
+ }
+}
+
static void ceph_osd_data_release(struct ceph_osd_data *osd_data) {
if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
@@ -1887,17 +1906,19 @@ bad:
static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
struct ceph_osd_data *osd_data)
{
+ u64 length = ceph_osd_data_length(osd_data);
+
if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
- BUG_ON(osd_data->length > (u64) SIZE_MAX);
- if (osd_data->length)
+ BUG_ON(length > (u64) SIZE_MAX);
+ if (length)
ceph_msg_data_set_pages(msg, osd_data->pages,
- osd_data->length, osd_data->alignment);
+ length, osd_data->alignment);
} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
- BUG_ON(!osd_data->pagelist->length);
+ BUG_ON(!length);
ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
#ifdef CONFIG_BLOCK
} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
- ceph_msg_data_set_bio(msg, osd_data->bio, osd_data->bio_length);
+ ceph_msg_data_set_bio(msg, osd_data->bio, length);
#endif
} else {
BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);

Signature · 1 line
1.7.9.5
Reviewed-by on v1 5/20
#10 lore raw

Re: [PATCH 05/20] libceph: define ceph_osd_data_length()

Quoted 65 lines · Alex Elder

One more osd data helper, which returns the length of the data item, regardless of its type.

---
  net/ceph/osd_client.c |   31 ++++++++++++++++++++++++++-----
  1 file changed, 26 insertions(+), 5 deletions(-)
net/ceph/osd_client.c +19 -0
index df0f856..110cd83 100644
@@ -117,6 +117,25 @@ void ceph_osd_data_bio_init(struct ceph_osd_data
 *osd_data,
  EXPORT_SYMBOL(ceph_osd_data_bio_init);
  #endif /* CONFIG_BLOCK */
 
+static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data)
+{
+	switch (osd_data->type) {
+	case CEPH_OSD_DATA_TYPE_NONE:
+		return 0;
+	case CEPH_OSD_DATA_TYPE_PAGES:
+		return osd_data->length;
+	case CEPH_OSD_DATA_TYPE_PAGELIST:
+		return (u64)osd_data->pagelist->length;
+#ifdef CONFIG_BLOCK
+	case CEPH_OSD_DATA_TYPE_BIO:
+		return (u64)osd_data->bio_length;
+#endif /* CONFIG_BLOCK */
+	default:
+		WARN(true, "unrecognized data type %d\n", (int)osd_data->type);
+		return 0;
+	}
+}
+
  static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
  {

if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)

net/ceph/osd_client.c +7 -5
@@ -1887,17 +1906,19 @@ bad:
  static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
  				struct ceph_osd_data *osd_data)
  {
+	u64 length = ceph_osd_data_length(osd_data);
+
  	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
-		BUG_ON(osd_data->length > (u64) SIZE_MAX);
-		if (osd_data->length)
+		BUG_ON(length > (u64) SIZE_MAX);
+		if (length)
  			ceph_msg_data_set_pages(msg, osd_data->pages,
-				osd_data->length, osd_data->alignment);
+					length, osd_data->alignment);
  	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
-		BUG_ON(!osd_data->pagelist->length);
+		BUG_ON(!length);
  		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
  #ifdef CONFIG_BLOCK
  	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
-		ceph_msg_data_set_bio(msg, osd_data->bio, osd_data->bio_length);
+		ceph_msg_data_set_bio(msg, osd_data->bio, length);
  #endif
  	} else {
  		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
author
#11 lore raw

PATCH 6/20 libceph: a few more osd data cleanups

In reply to Alex Elder

These are very small changes that make use osd_data local pointers as shorthands for structures being operated on.

---
 fs/ceph/addr.c |   30 +++++++++++++++++-------------
 1 file changed, 17 insertions(+), 13 deletions(-)
fs/ceph/addr.c +0 -0
index 8507389..c46b9d3 100644
@@ -236,6 +236,7 @@ static int ceph_readpage(struct file *filp, struct

page *page)
static void finish_read(struct ceph_osd_request *req, struct ceph_msg *msg) {
struct inode *inode = req->r_inode;
+ struct ceph_osd_data *osd_data;
int rc = req->r_result;
int bytes = le32_to_cpu(msg->hdr.data_len);
int num_pages;
@@ -244,11 +245,12 @@ static void finish_read(struct ceph_osd_request *req, struct ceph_msg *msg)
dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);

/* unlock all pages, zeroing any data we didn't read */
- BUG_ON(req->r_data_in.type != CEPH_OSD_DATA_TYPE_PAGES);
- num_pages = calc_pages_for((u64)req->r_data_in.alignment,
- (u64)req->r_data_in.length);
+ osd_data = &req->r_data_in;
+ BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+ num_pages = calc_pages_for((u64)osd_data->alignment,
+ (u64)osd_data->length);
for (i = 0; i < num_pages; i++) {
- struct page *page = req->r_data_in.pages[i];
+ struct page *page = osd_data->pages[i];

if (bytes < (int)PAGE_CACHE_SIZE) {
/* zero (remainder of) page */
@@ -263,7 +265,7 @@ static void finish_read(struct ceph_osd_request *req, struct ceph_msg *msg)
page_cache_release(page);
bytes -= PAGE_CACHE_SIZE;
}
- kfree(req->r_data_in.pages);
+ kfree(osd_data->pages);
}

static void ceph_unlock_page_vector(struct page **pages, int num_pages)
@@ -557,6 +559,7 @@ static void writepages_finish(struct ceph_osd_request *req,
{
struct inode *inode = req->r_inode;
struct ceph_inode_info *ci = ceph_inode(inode);
+ struct ceph_osd_data *osd_data;
unsigned wrote;
struct page *page;
int num_pages;
@@ -569,9 +572,10 @@ static void writepages_finish(struct ceph_osd_request *req,
long writeback_stat;
unsigned issued = ceph_caps_issued(ci);

- BUG_ON(req->r_data_out.type != CEPH_OSD_DATA_TYPE_PAGES);
- num_pages = calc_pages_for((u64)req->r_data_out.alignment,
- (u64)req->r_data_out.length);
+ osd_data = &req->r_data_out;
+ BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+ num_pages = calc_pages_for((u64)osd_data->alignment,
+ (u64)osd_data->length);
if (rc >= 0) {
/*
* Assume we wrote the pages we originally sent. The
@@ -589,7 +593,7 @@ static void writepages_finish(struct ceph_osd_request *req,

/* clean all pages */
for (i = 0; i < num_pages; i++) {
- page = req->r_data_out.pages[i];
+ page = osd_data->pages[i];
BUG_ON(!page);
WARN_ON(!PageUptodate(page));

@@ -620,12 +624,12 @@ static void writepages_finish(struct ceph_osd_request *req,
dout("%p wrote+cleaned %d pages\n", inode, wrote); ceph_put_wrbuffer_cap_refs(ci, num_pages, snapc);

- ceph_release_pages(req->r_data_out.pages, num_pages);
- if (req->r_data_out.pages_from_pool)
- mempool_free(req->r_data_out.pages,
+ ceph_release_pages(osd_data->pages, num_pages);
+ if (osd_data->pages_from_pool)
+ mempool_free(osd_data->pages,
ceph_sb_to_client(inode->i_sb)->wb_pagevec_pool);
else
- kfree(req->r_data_out.pages);
+ kfree(osd_data->pages);
ceph_osdc_put_request(req);
}

Signature · 1 line
1.7.9.5
Reviewed-by on v1 6/20
#12 lore raw

Re: [PATCH 06/20] libceph: a few more osd data cleanups

Quoted 101 lines · Alex Elder

These are very small changes that make use osd_data local pointers as shorthands for structures being operated on.

---
  fs/ceph/addr.c |   30 +++++++++++++++++-------------
  1 file changed, 17 insertions(+), 13 deletions(-)
fs/ceph/addr.c +1 -0
index 8507389..c46b9d3 100644
@@ -236,6 +236,7 @@ static int ceph_readpage(struct file *filp, struct
 page *page)
  static void finish_read(struct ceph_osd_request *req, struct ceph_msg *msg)
  {
  	struct inode *inode = req->r_inode;
+	struct ceph_osd_data *osd_data;
  	int rc = req->r_result;
  	int bytes = le32_to_cpu(msg->hdr.data_len);

int num_pages;

fs/ceph/addr.c +5 -4
@@ -244,11 +245,12 @@ static void finish_read(struct ceph_osd_request
 *req, struct ceph_msg *msg)
  	dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);
 
  	/* unlock all pages, zeroing any data we didn't read */
-	BUG_ON(req->r_data_in.type != CEPH_OSD_DATA_TYPE_PAGES);
-	num_pages = calc_pages_for((u64)req->r_data_in.alignment,
-					(u64)req->r_data_in.length);
+	osd_data = &req->r_data_in;
+	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+	num_pages = calc_pages_for((u64)osd_data->alignment,
+					(u64)osd_data->length);
  	for (i = 0; i < num_pages; i++) {
-		struct page *page = req->r_data_in.pages[i];
+		struct page *page = osd_data->pages[i];
 
  		if (bytes < (int)PAGE_CACHE_SIZE) {

/* zero (remainder of) page */

fs/ceph/addr.c +1 -1
@@ -263,7 +265,7 @@ static void finish_read(struct ceph_osd_request
 *req, struct ceph_msg *msg)
  		page_cache_release(page);
  		bytes -= PAGE_CACHE_SIZE;
  	}
-	kfree(req->r_data_in.pages);
+	kfree(osd_data->pages);
  }
 

static void ceph_unlock_page_vector(struct page **pages, int num_pages)

fs/ceph/addr.c +1 -0
@@ -557,6 +559,7 @@ static void writepages_finish(struct
 ceph_osd_request *req,
  {
  	struct inode *inode = req->r_inode;
  	struct ceph_inode_info *ci = ceph_inode(inode);
+	struct ceph_osd_data *osd_data;
  	unsigned wrote;
  	struct page *page;

int num_pages;

fs/ceph/addr.c +4 -3
@@ -569,9 +572,10 @@ static void writepages_finish(struct
 ceph_osd_request *req,
  	long writeback_stat;
  	unsigned issued = ceph_caps_issued(ci);
 
-	BUG_ON(req->r_data_out.type != CEPH_OSD_DATA_TYPE_PAGES);
-	num_pages = calc_pages_for((u64)req->r_data_out.alignment,
-					(u64)req->r_data_out.length);
+	osd_data = &req->r_data_out;
+	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
+	num_pages = calc_pages_for((u64)osd_data->alignment,
+					(u64)osd_data->length);
  	if (rc >= 0) {
  		/*

* Assume we wrote the pages we originally sent. The

fs/ceph/addr.c +5 -5
@@ -589,7 +593,7 @@ static void writepages_finish(struct
 ceph_osd_request *req,
 
  	/* clean all pages */
  	for (i = 0; i < num_pages; i++) {
-		page = req->r_data_out.pages[i];
+		page = osd_data->pages[i];
  		BUG_ON(!page);
  		WARN_ON(!PageUptodate(page));
@@ -620,12 +624,12 @@ static void writepages_finish(struct
 ceph_osd_request *req,
  	dout("%p wrote+cleaned %d pages\n", inode, wrote);
  	ceph_put_wrbuffer_cap_refs(ci, num_pages, snapc);
 
-	ceph_release_pages(req->r_data_out.pages, num_pages);
-	if (req->r_data_out.pages_from_pool)
-		mempool_free(req->r_data_out.pages,
+	ceph_release_pages(osd_data->pages, num_pages);
+	if (osd_data->pages_from_pool)
+		mempool_free(osd_data->pages,
  			     ceph_sb_to_client(inode->i_sb)->wb_pagevec_pool);
  	else
-		kfree(req->r_data_out.pages);
+		kfree(osd_data->pages);
  	ceph_osdc_put_request(req);
  }
author
#13 lore raw

PATCH 7/20 rbd: define rbd_osd_req_format_op()

In reply to Alex Elder

Define rbd_osd_req_format_op(), which encapsulates formatting an osd op into an object request's osd request message. Only one op is supported right now.

Stop calling ceph_osdc_build_request() in rbd_osd_req_create(). Instead, call rbd_osd_req_format_op() in each of the callers of rbd_osd_req_create().

This is to prepare for the next patch, in which the source ops for an osd request will be held in the osd request itself. Because of that, we won't have the source op to work with until after the request is created, so we can't format the op until then.

This an the next patch resolve:

    http://tracker.ceph.com/issues/4656
---
 drivers/block/rbd.c |   98
++++++++++++++++++++++++++++-----------------------
 1 file changed, 53 insertions(+), 45 deletions(-)
drivers/block/rbd.c +0 -0
index a9d88a0..fc41675 100644
@@ -1311,29 +1311,47 @@ static void rbd_osd_req_callback(struct

ceph_osd_request *osd_req,
rbd_obj_request_complete(obj_request);
}

+static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
+ bool write_request,
+ struct ceph_osd_req_op *op)
+{
+ struct rbd_img_request *img_request = obj_request->img_request;
+ struct ceph_snap_context *snapc = NULL;
+ u64 snap_id = CEPH_NOSNAP;
+ struct timespec *mtime = NULL;
+ struct timespec now;
+
+ rbd_assert(obj_request->osd_req != NULL);
+
+ if (write_request) {
+ now = CURRENT_TIME;
+ mtime = &now;
+ if (img_request)
+ snapc = img_request->snapc;
+ } else if (img_request) {
+ snap_id = img_request->snap_id;
+ }
+
+ ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
+ 1, op, snapc, snap_id, mtime);
+}
+
static struct ceph_osd_request *rbd_osd_req_create(
struct rbd_device *rbd_dev, bool write_request,
- struct rbd_obj_request *obj_request,
- struct ceph_osd_req_op *op)
+ struct rbd_obj_request *obj_request)
{
struct rbd_img_request *img_request = obj_request->img_request; struct ceph_snap_context *snapc = NULL;
struct ceph_osd_client *osdc;
struct ceph_osd_request *osd_req;
struct ceph_osd_data *osd_data;
- struct timespec now;
- struct timespec *mtime;
- u64 snap_id = CEPH_NOSNAP;
u64 offset = obj_request->offset;
- u64 length = obj_request->length;

if (img_request) {
rbd_assert(img_request->write_request == write_request); if (img_request->write_request)
snapc = img_request->snapc;
- else
- snap_id = img_request->snap_id;
}

/* Allocate and initialize the request, for the single op */
@@ -1360,16 +1378,10 @@ static struct ceph_osd_request *rbd_osd_req_create(
break;
}

- if (write_request) {
+ if (write_request)
osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
- now = CURRENT_TIME;
- mtime = &now;
- } else {
+ else
osd_req->r_flags = CEPH_OSD_FLAG_READ;
- mtime = NULL; /* not needed for reads */
- offset = 0; /* These are not used... */
- length = 0; /* ...for osd read requests */
- }

osd_req->r_callback = rbd_osd_req_callback; osd_req->r_priv = obj_request;
@@ -1380,11 +1392,6 @@ static struct ceph_osd_request *rbd_osd_req_create(

osd_req->r_file_layout = rbd_dev->layout; /* struct */

- /* osd_req will get its own reference to snapc (if non-null) */
-
- ceph_osdc_build_request(osd_req, offset, 1, op,
- snapc, snap_id, mtime);
-
return osd_req;
}

@@ -1538,6 +1545,7 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
struct rbd_device *rbd_dev = img_request->rbd_dev; struct rbd_obj_request *obj_request = NULL; struct rbd_obj_request *next_obj_request;
+ bool write_request = img_request->write_request;
unsigned int bio_offset;
u64 image_offset;
u64 resid;
@@ -1545,8 +1553,7 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,

dout("%s: img %p bio %p\n", __func__, img_request, bio_list);

- opcode = img_request->write_request ? CEPH_OSD_OP_WRITE
- : CEPH_OSD_OP_READ;
+ opcode = write_request ? CEPH_OSD_OP_WRITE : CEPH_OSD_OP_READ;
bio_offset = 0;
image_offset = img_request->offset; rbd_assert(image_offset == bio_list->bi_sector << SECTOR_SHIFT);
@@ -1579,17 +1586,14 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
if (!obj_request->bio_list)
goto out_partial;

- /*
- * Build up the op to use in building the osd
- * request. Note that the contents of the op are
- * copied by rbd_osd_req_create().
- */
- osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
obj_request->osd_req = rbd_osd_req_create(rbd_dev,
- img_request->write_request,
- obj_request, &op);
+ write_request, obj_request);
if (!obj_request->osd_req)
goto out_partial;
+
+ osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
+ rbd_osd_req_format_op(obj_request, write_request, &op);
+
/* status and version are initially zero-filled */

rbd_img_obj_request_add(img_request, obj_request);
@@ -1699,12 +1703,13 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,
return -ENOMEM;

ret = -ENOMEM;
- osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
- obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
- obj_request, &op);
+ obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
if (!obj_request->osd_req)
goto out;

+ osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+ rbd_osd_req_format_op(obj_request, false, &op);
+
osdc = &rbd_dev->rbd_client->client->osdc; obj_request->callback = rbd_obj_request_put;
ret = rbd_obj_request_submit(osdc, obj_request);
@@ -1763,13 +1768,14 @@ static int rbd_dev_header_watch_sync(struct rbd_device *rbd_dev, int start)
if (!obj_request)
goto out_cancel;

+ obj_request->osd_req = rbd_osd_req_create(rbd_dev, true, obj_request);
+ if (!obj_request->osd_req)
+ goto out_cancel;
+
osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
rbd_dev->watch_event->cookie, rbd_dev->header.obj_version, start);
- obj_request->osd_req = rbd_osd_req_create(rbd_dev, true,
- obj_request, &op);
- if (!obj_request->osd_req)
- goto out_cancel;
+ rbd_osd_req_format_op(obj_request, true, &op);

if (start)
ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
@@ -1855,13 +1861,14 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
obj_request->pages = pages;
obj_request->page_count = page_count;

- osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
- outbound, outbound_size);
- obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
- obj_request, &op);
+ obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
if (!obj_request->osd_req)
goto out;

+ osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
+ outbound, outbound_size);
+ rbd_osd_req_format_op(obj_request, false, &op);
+
osdc = &rbd_dev->rbd_client->client->osdc;
ret = rbd_obj_request_submit(osdc, obj_request);
if (ret)
@@ -2060,12 +2067,13 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
obj_request->pages = pages;
obj_request->page_count = page_count;

- osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
- obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
- obj_request, &op);
+ obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
if (!obj_request->osd_req)
goto out;

+ osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+ rbd_osd_req_format_op(obj_request, false, &op);
+
osdc = &rbd_dev->rbd_client->client->osdc; ret = rbd_obj_request_submit(osdc, obj_request); if (ret)

Signature · 1 line
1.7.9.5
Reviewed-by on v1 7/20
#14 lore raw

Re: [PATCH 07/20] rbd: define rbd_osd_req_format_op()

Quoted 235 lines · Alex Elder

Define rbd_osd_req_format_op(), which encapsulates formatting an osd op into an object request's osd request message. Only one op is supported right now.

Stop calling ceph_osdc_build_request() in rbd_osd_req_create(). Instead, call rbd_osd_req_format_op() in each of the callers of rbd_osd_req_create().

This is to prepare for the next patch, in which the source ops for an osd request will be held in the osd request itself. Because of that, we won't have the source op to work with until after the request is created, so we can't format the op until then.

This an the next patch resolve:

     http://tracker.ceph.com/issues/4656
---
  drivers/block/rbd.c |   98
++++++++++++++++++++++++++++-----------------------
  1 file changed, 53 insertions(+), 45 deletions(-)
drivers/block/rbd.c +26 -8
index a9d88a0..fc41675 100644
@@ -1311,29 +1311,47 @@ static void rbd_osd_req_callback(struct
 ceph_osd_request *osd_req,
  		rbd_obj_request_complete(obj_request);
  }
 
+static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
+					bool write_request,
+					struct ceph_osd_req_op *op)
+{
+	struct rbd_img_request *img_request = obj_request->img_request;
+	struct ceph_snap_context *snapc = NULL;
+	u64 snap_id = CEPH_NOSNAP;
+	struct timespec *mtime = NULL;
+	struct timespec now;
+
+	rbd_assert(obj_request->osd_req != NULL);
+
+	if (write_request) {
+		now = CURRENT_TIME;
+		mtime = &now;
+		if (img_request)
+			snapc = img_request->snapc;
+	} else if (img_request) {
+		snap_id = img_request->snap_id;
+	}
+
+	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
+			1, op, snapc, snap_id, mtime);
+}
+
  static struct ceph_osd_request *rbd_osd_req_create(
  					struct rbd_device *rbd_dev,
  					bool write_request,
-					struct rbd_obj_request *obj_request,
-					struct ceph_osd_req_op *op)
+					struct rbd_obj_request *obj_request)
  {
  	struct rbd_img_request *img_request = obj_request->img_request;
  	struct ceph_snap_context *snapc = NULL;
  	struct ceph_osd_client *osdc;
  	struct ceph_osd_request *osd_req;
  	struct ceph_osd_data *osd_data;
-	struct timespec now;
-	struct timespec *mtime;
-	u64 snap_id = CEPH_NOSNAP;
  	u64 offset = obj_request->offset;
-	u64 length = obj_request->length;
 
  	if (img_request) {
  		rbd_assert(img_request->write_request == write_request);
  		if (img_request->write_request)
  			snapc = img_request->snapc;
-		else
-			snap_id = img_request->snap_id;
  	}
 

/* Allocate and initialize the request, for the single op */

drivers/block/rbd.c +3 -13
@@ -1360,16 +1378,10 @@ static struct ceph_osd_request *rbd_osd_req_create(
  		break;
  	}
 
-	if (write_request) {
+	if (write_request)
  		osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
-		now = CURRENT_TIME;
-		mtime = &now;
-	} else {
+	else
  		osd_req->r_flags = CEPH_OSD_FLAG_READ;
-		mtime = NULL;	/* not needed for reads */
-		offset = 0;	/* These are not used... */
-		length = 0;	/* ...for osd read requests */
-	}
 
  	osd_req->r_callback = rbd_osd_req_callback;
  	osd_req->r_priv = obj_request;
@@ -1380,11 +1392,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
 
  	osd_req->r_file_layout = rbd_dev->layout;	/* struct */
 
-	/* osd_req will get its own reference to snapc (if non-null) */
-
-	ceph_osdc_build_request(osd_req, offset, 1, op,
-				snapc, snap_id, mtime);
-
  	return osd_req;
  }
 
@@ -1538,6 +1545,7 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  	struct rbd_device *rbd_dev = img_request->rbd_dev;
  	struct rbd_obj_request *obj_request = NULL;
  	struct rbd_obj_request *next_obj_request;
+	bool write_request = img_request->write_request;
  	unsigned int bio_offset;
  	u64 image_offset;

u64 resid;

drivers/block/rbd.c +1 -2
@@ -1545,8 +1553,7 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
 
  	dout("%s: img %p bio %p\n", __func__, img_request, bio_list);
 
-	opcode = img_request->write_request ? CEPH_OSD_OP_WRITE
-					      : CEPH_OSD_OP_READ;
+	opcode = write_request ? CEPH_OSD_OP_WRITE : CEPH_OSD_OP_READ;
  	bio_offset = 0;
  	image_offset = img_request->offset;
  	rbd_assert(image_offset == bio_list->bi_sector << SECTOR_SHIFT);
drivers/block/rbd.c +5 -8
@@ -1579,17 +1586,14 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  		if (!obj_request->bio_list)
  			goto out_partial;
 
-		/*
-		 * Build up the op to use in building the osd
-		 * request.  Note that the contents of the op are
-		 * copied by rbd_osd_req_create().
-		 */
-		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
  		obj_request->osd_req = rbd_osd_req_create(rbd_dev,
-						img_request->write_request,
-						obj_request, &op);
+						write_request, obj_request);
  		if (!obj_request->osd_req)
  			goto out_partial;
+
+		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
+		rbd_osd_req_format_op(obj_request, write_request, &op);
+
  		/* status and version are initially zero-filled */
 
  		rbd_img_obj_request_add(img_request, obj_request);
drivers/block/rbd.c +4 -3
@@ -1699,12 +1703,13 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
  		return -ENOMEM;
 
  	ret = -ENOMEM;
-	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
-						obj_request, &op);
+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
  	if (!obj_request->osd_req)
  		goto out;
 
+	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+	rbd_osd_req_format_op(obj_request, false, &op);
+
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	obj_request->callback = rbd_obj_request_put;
  	ret = rbd_obj_request_submit(osdc, obj_request);
drivers/block/rbd.c +5 -4
@@ -1763,13 +1768,14 @@ static int rbd_dev_header_watch_sync(struct
 rbd_device *rbd_dev, int start)
  	if (!obj_request)
  		goto out_cancel;
 
+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, true, obj_request);
+	if (!obj_request->osd_req)
+		goto out_cancel;
+
  	osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
  				rbd_dev->watch_event->cookie,
  				rbd_dev->header.obj_version, start);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, true,
-							obj_request, &op);
-	if (!obj_request->osd_req)
-		goto out_cancel;
+	rbd_osd_req_format_op(obj_request, true, &op);
 
  	if (start)
  		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
drivers/block/rbd.c +5 -4
@@ -1855,13 +1861,14 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  	obj_request->pages = pages;
  	obj_request->page_count = page_count;
 
-	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
-					outbound, outbound_size);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
-						obj_request, &op);
+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
  	if (!obj_request->osd_req)
  		goto out;
 
+	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
+					outbound, outbound_size);
+	rbd_osd_req_format_op(obj_request, false, &op);
+
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	ret = rbd_obj_request_submit(osdc, obj_request);

if (ret)

drivers/block/rbd.c +4 -3
@@ -2060,12 +2067,13 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  	obj_request->pages = pages;
  	obj_request->page_count = page_count;
 
-	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
-	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
-						obj_request, &op);
+	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request);
  	if (!obj_request->osd_req)
  		goto out;
 
+	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+	rbd_osd_req_format_op(obj_request, false, &op);
+
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	ret = rbd_obj_request_submit(osdc, obj_request);

if (ret)

author
#15 lore raw

PATCH 8/20 libceph: keep source rather than message osd op array

In reply to Alex Elder

An osd request keeps a pointer to the osd operations (ops) array that it builds in its request message.

In order to allow each op in the array to have its own distinct data, we will need to keep track of each op's data, and that information does not go over the wire.

As long as we're tracking the data we might as well just track the entire (source) op definition for each of the ops. And if we're doing that, we'll have no more need to keep a pointer to the wire-encoded version.

This patch makes the array of source ops be kept with the osd request structure, and uses that instead of the version encoded in the message in places where that was previously used. The array will be embedded in the request structure, and the maximum number of ops we ever actually use is currently 2. So reduce CEPH_OSD_MAX_OP to 2 to reduce the size of the structure.

The result of doing this sort of ripples back up, and as a result various function parameters and local variables become unnecessary.

Make r_num_ops be unsigned, and move the definition of struct ceph_osd_req_op earlier to ensure it's defined where needed.

It does not yet add per-op data, that's coming soon.

This resolves:

    http://tracker.ceph.com/issues/4656
---
 drivers/block/rbd.c             |   42 ++++++++++++-----------
 fs/ceph/addr.c                  |   21 +++++-------
 fs/ceph/file.c                  |    6 ++--
 include/linux/ceph/osd_client.h |   70
+++++++++++++++++++--------------------
 net/ceph/debugfs.c              |    4 +--
 net/ceph/osd_client.c           |   53 ++++++++++++++---------------
 6 files changed, 97 insertions(+), 99 deletions(-)
drivers/block/rbd.c +0 -0
index fc41675..e6e2191 100644
@@ -1285,7 +1285,7 @@ static void rbd_osd_req_callback(struct

ceph_osd_request *osd_req,
*/
obj_request->xferred = osd_req->r_reply_op_len[0]; rbd_assert(obj_request->xferred < (u64) UINT_MAX);
- opcode = osd_req->r_request_ops[0].op;
+ opcode = osd_req->r_ops[0].op;
switch (opcode) {
case CEPH_OSD_OP_READ:
rbd_osd_read_callback(obj_request);
@@ -1312,8 +1312,7 @@ static void rbd_osd_req_callback(struct ceph_osd_request *osd_req,
}

static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
- bool write_request,
- struct ceph_osd_req_op *op)
+ bool write_request)
{
struct rbd_img_request *img_request = obj_request->img_request; struct ceph_snap_context *snapc = NULL;
@@ -1333,7 +1332,7 @@ static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
}

ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
- 1, op, snapc, snap_id, mtime);
+ snapc, snap_id, mtime);
}

static struct ceph_osd_request *rbd_osd_req_create(
@@ -1562,7 +1561,7 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
while (resid) {
const char *object_name;
unsigned int clone_size;
- struct ceph_osd_req_op op;
+ struct ceph_osd_req_op *op;
u64 offset;
u64 length;

@@ -1591,8 +1590,9 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
if (!obj_request->osd_req)
goto out_partial;

- osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
- rbd_osd_req_format_op(obj_request, write_request, &op);
+ op = &obj_request->osd_req->r_ops[0];
+ osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
+ rbd_osd_req_format_op(obj_request, write_request);

/* status and version are initially zero-filled */

@@ -1693,7 +1693,7 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,
u64 ver, u64 notify_id)
{
struct rbd_obj_request *obj_request;
- struct ceph_osd_req_op op;
+ struct ceph_osd_req_op *op;
struct ceph_osd_client *osdc;
int ret;

@@ -1707,8 +1707,9 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
- rbd_osd_req_format_op(obj_request, false, &op);
+ op = &obj_request->osd_req->r_ops[0];
+ osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+ rbd_osd_req_format_op(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc; obj_request->callback = rbd_obj_request_put;
@@ -1748,7 +1749,7 @@ static int rbd_dev_header_watch_sync(struct rbd_device *rbd_dev, int start)
{
struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc; struct rbd_obj_request *obj_request;
- struct ceph_osd_req_op op;
+ struct ceph_osd_req_op *op;
int ret;

rbd_assert(start ^ !!rbd_dev->watch_event);
@@ -1772,10 +1773,11 @@ static int rbd_dev_header_watch_sync(struct rbd_device *rbd_dev, int start)
if (!obj_request->osd_req)
goto out_cancel;

- osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
+ op = &obj_request->osd_req->r_ops[0];
+ osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
rbd_dev->watch_event->cookie, rbd_dev->header.obj_version, start);
- rbd_osd_req_format_op(obj_request, true, &op);
+ rbd_osd_req_format_op(obj_request, true);

if (start)
ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
@@ -1835,7 +1837,7 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
{
struct rbd_obj_request *obj_request;
struct ceph_osd_client *osdc;
- struct ceph_osd_req_op op;
+ struct ceph_osd_req_op *op;
struct page **pages;
u32 page_count;
int ret;
@@ -1865,9 +1867,10 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
+ op = &obj_request->osd_req->r_ops[0];
+ osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
outbound, outbound_size);
- rbd_osd_req_format_op(obj_request, false, &op);
+ rbd_osd_req_format_op(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc;
ret = rbd_obj_request_submit(osdc, obj_request);
@@ -2045,8 +2048,8 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
char *buf, u64 *version)

{
- struct ceph_osd_req_op op;
struct rbd_obj_request *obj_request;
+ struct ceph_osd_req_op *op;
struct ceph_osd_client *osdc;
struct page **pages = NULL;
u32 page_count;
@@ -2071,8 +2074,9 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
- rbd_osd_req_format_op(obj_request, false, &op);
+ op = &obj_request->osd_req->r_ops[0];
+ osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+ rbd_osd_req_format_op(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc; ret = rbd_obj_request_submit(osdc, obj_request);

fs/ceph/addr.c +0 -0
index c46b9d3..12bbba7 100644
@@ -288,7 +288,6 @@ static int start_read(struct inode *inode, struct

list_head *page_list, int max)
struct page *page = list_entry(page_list->prev, struct page, lru); struct ceph_vino vino;
struct ceph_osd_request *req;
- struct ceph_osd_req_op op;
u64 off;
u64 len;
int i;
@@ -314,7 +313,7 @@ static int start_read(struct inode *inode, struct list_head *page_list, int max)
off, len);
vino = ceph_vino(inode);
req = ceph_osdc_new_request(osdc, &ci->i_layout, vino, off, &len,
- 1, &op, CEPH_OSD_OP_READ,
+ 1, CEPH_OSD_OP_READ,
CEPH_OSD_FLAG_READ, NULL, ci->i_truncate_seq, ci->i_truncate_size, false);
@@ -349,7 +348,7 @@ static int start_read(struct inode *inode, struct list_head *page_list, int max)
req->r_callback = finish_read;
req->r_inode = inode;

- ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
+ ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);

dout("start_read %p starting %p %lld~%lld\n", inode, req, off, len); ret = ceph_osdc_start_request(osdc, req, false);
@@ -567,7 +566,7 @@ static void writepages_finish(struct ceph_osd_request *req,
struct ceph_snap_context *snapc = req->r_snapc;
struct address_space *mapping = inode->i_mapping;
int rc = req->r_result;
- u64 bytes = le64_to_cpu(req->r_request_ops[0].extent.length);
+ u64 bytes = req->r_ops[0].extent.length;
struct ceph_fs_client *fsc = ceph_inode_to_client(inode); long writeback_stat;
unsigned issued = ceph_caps_issued(ci);
@@ -635,8 +634,7 @@ static void writepages_finish(struct ceph_osd_request *req,

static struct ceph_osd_request * ceph_writepages_osd_request(struct inode *inode, u64 offset, u64 *len,
- struct ceph_snap_context *snapc,
- int num_ops, struct ceph_osd_req_op *ops)
+ struct ceph_snap_context *snapc, int num_ops)
{
struct ceph_fs_client *fsc;
struct ceph_inode_info *ci;
@@ -648,7 +646,7 @@ ceph_writepages_osd_request(struct inode *inode, u64 offset, u64 *len,
/* BUG_ON(vino.snap != CEPH_NOSNAP); */

return ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
- vino, offset, len, num_ops, ops, CEPH_OSD_OP_WRITE,
+ vino, offset, len, num_ops, CEPH_OSD_OP_WRITE,
CEPH_OSD_FLAG_WRITE|CEPH_OSD_FLAG_ONDISK, snapc, ci->i_truncate_seq, ci->i_truncate_size, true);
}
@@ -738,7 +736,6 @@ retry:
last_snapc = snapc;

while (!done && index <= end) {
- struct ceph_osd_req_op ops[2];
int num_ops = do_sync ? 2 : 1;
struct ceph_vino vino;
unsigned i;
@@ -846,7 +843,7 @@ get_more_pages:
len = wsize;
req = ceph_writepages_osd_request(inode,
offset, &len, snapc,
- num_ops, ops);
+ num_ops);

if (IS_ERR(req)) {
rc = PTR_ERR(req);
@@ -927,11 +924,11 @@ get_more_pages:

/* Update the write op length in case we changed it */

- osd_req_op_extent_update(&ops[0], len);
+ osd_req_op_extent_update(&req->r_ops[0], len);

vino = ceph_vino(inode);
- ceph_osdc_build_request(req, offset, num_ops, ops,
- snapc, vino.snap, &inode->i_mtime);
+ ceph_osdc_build_request(req, offset, snapc, vino.snap,
+ &inode->i_mtime);

rc = ceph_osdc_start_request(&fsc->client->osdc, req, true); BUG_ON(rc);

fs/ceph/file.c +0 -0
index 2f2d0a1..a1d85a8 100644
@@ -478,7 +478,6 @@ static ssize_t ceph_sync_write(struct file *file,

const char __user *data,
struct ceph_snap_context *snapc;
struct ceph_vino vino;
struct ceph_osd_request *req;
- struct ceph_osd_req_op ops[2];
int num_ops = 1;
struct page **pages;
int num_pages;
@@ -534,7 +533,7 @@ more:
snapc = ci->i_snap_realm->cached_context;
vino = ceph_vino(inode);
req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
- vino, pos, &len, num_ops, ops,
+ vino, pos, &len, num_ops,
CEPH_OSD_OP_WRITE, flags, snapc, ci->i_truncate_seq, ci->i_truncate_size, false);
@@ -579,8 +578,7 @@ more:
false, own_pages);

/* BUG_ON(vino.snap != CEPH_NOSNAP); */
- ceph_osdc_build_request(req, pos, num_ops, ops,
- snapc, vino.snap, &mtime);
+ ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);

ret = ceph_osdc_start_request(&fsc->client->osdc, req, false); if (!ret) {

(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index af60dac..f4c1a2a 100644

include/linux/ceph/osd_client.h +33 -32

90 lines not on this page · raw message

*ceph_osdc_alloc_request(struct ceph_osd_client *
gfp_t gfp_flags);

extern void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
- unsigned int num_ops,
- struct ceph_osd_req_op *src_ops,
struct ceph_snap_context *snapc, u64 snap_id,
struct timespec *mtime);
@@ -259,8 +258,7 @@ extern struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *,
struct ceph_file_layout *layout, struct ceph_vino vino, u64 offset, u64 *len,

-				      int num_ops, struct ceph_osd_req_op *ops,
-				      int opcode, int flags,
+				      int num_ops, int opcode, int flags,

struct ceph_snap_context *snapc, u32 truncate_seq, u64 truncate_size, bool use_mempool);

net/ceph/debugfs.c +2 -2
index 00d051f..83661cd 100644
@@ -123,8 +123,8 @@ static int osdc_show(struct seq_file *s, void *pp)
 	mutex_lock(&osdc->request_mutex);
 	for (p = rb_first(&osdc->requests); p; p = rb_next(p)) {
 		struct ceph_osd_request *req;
+		unsigned int i;
 		int opcode;
-		int i;
 
 		req = rb_entry(p, struct ceph_osd_request, r_node);
 
@@ -142,7 +142,7 @@ static int osdc_show(struct seq_file *s, void *pp)
 			seq_printf(s, "\t");
 
 		for (i = 0; i < req->r_num_ops; i++) {
-			opcode = le16_to_cpu(req->r_request_ops[i].op);
+			opcode = req->r_ops[i].op;
 			seq_printf(s, "\t%s", ceph_osd_op_name(opcode));
 		}
 
net/ceph/osd_client.c +0 -0
index 110cd83..ed4493d 100644
@@ -186,6 +186,9 @@ struct ceph_osd_request

*ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
struct ceph_msg *msg;
size_t msg_size;

+ BUILD_BUG_ON(CEPH_OSD_MAX_OP > U16_MAX);
+ BUG_ON(num_ops > CEPH_OSD_MAX_OP);
+
msg_size = 4 + 4 + 8 + 8 + 4+8; msg_size += 2 + 4 + 8 + 4 + 4; /* oloc */
msg_size += 1 + 8 + 4 + 4; /* pg_t */
@@ -207,6 +210,7 @@ struct ceph_osd_request *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,

req->r_osdc = osdc;
req->r_mempool = use_mempool;
+ req->r_num_ops = num_ops;

kref_init(&req->r_kref); init_completion(&req->r_completion);
@@ -418,12 +422,14 @@ void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
EXPORT_SYMBOL(osd_req_op_watch_init);

static u64 osd_req_encode_op(struct ceph_osd_request *req,

-			      struct ceph_osd_op *dst,
-			      struct ceph_osd_req_op *src)
+			      struct ceph_osd_op *dst, unsigned int which)

{
+ struct ceph_osd_req_op *src;
u64 out_data_len = 0;
struct ceph_pagelist *pagelist;

+ BUG_ON(which >= req->r_num_ops);
+ src = &req->r_ops[which];
if (WARN_ON(!osd_req_opcode_valid(src->op))) {
pr_err("unrecognized osd opcode %d\n", src->op);

@@ -487,21 +493,17 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
* build new request AND message
*
*/
-void ceph_osdc_build_request(struct ceph_osd_request *req,

-			     u64 off, unsigned int num_ops,
-			     struct ceph_osd_req_op *src_ops,
-			     struct ceph_snap_context *snapc, u64 snap_id,
-			     struct timespec *mtime)

+void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
+ struct ceph_snap_context *snapc, u64 snap_id,
+ struct timespec *mtime)
{
struct ceph_msg *msg = req->r_request;
- struct ceph_osd_req_op *src_op;
void *p;
size_t msg_size;
int flags = req->r_flags;
u64 data_len;
- int i;
+ unsigned int i;

- req->r_num_ops = num_ops;
req->r_snapid = snap_id; req->r_snapc = ceph_get_snap_context(snapc);

@@ -541,12 +543,10 @@ void ceph_osdc_build_request(struct ceph_osd_request *req,
p += req->r_oid_len;

/* ops--can imply data */
- ceph_encode_16(&p, num_ops);
- src_op = src_ops;
- req->r_request_ops = p;
+ ceph_encode_16(&p, (u16)req->r_num_ops);
data_len = 0;
- for (i = 0; i < num_ops; i++, src_op++) {
- data_len += osd_req_encode_op(req, p, src_op);
+ for (i = 0; i < req->r_num_ops; i++) {
+ data_len += osd_req_encode_op(req, p, i);
p += sizeof(struct ceph_osd_op);
}

@@ -602,7 +602,6 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
struct ceph_file_layout *layout, struct ceph_vino vino, u64 off, u64 *plen, int num_ops,
- struct ceph_osd_req_op *ops,
int opcode, int flags, struct ceph_snap_context *snapc, u32 truncate_seq,
@@ -610,6 +609,7 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
bool use_mempool)
{
struct ceph_osd_request *req;
+ struct ceph_osd_req_op *op;
u64 objnum = 0;
u64 objoff = 0;
u64 objlen = 0;
@@ -623,6 +623,7 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
GFP_NOFS);
if (!req)
return ERR_PTR(-ENOMEM);
+
req->r_flags = flags;

/* calculate max write size */
@@ -642,7 +643,8 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
truncate_size = object_size;
}

- osd_req_op_extent_init(&ops[0], opcode, objoff, objlen,
+ op = &req->r_ops[0];
+ osd_req_op_extent_init(op, opcode, objoff, objlen,
truncate_size, truncate_seq);
/*
* A second op in the ops array means the caller wants to
@@ -650,7 +652,7 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
* osd will flush data quickly.
*/
if (num_ops > 1)
- osd_req_op_init(&ops[1], CEPH_OSD_OP_STARTSYNC);
+ osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);

req->r_file_layout = *layout; /* keep a copy */

@@ -1342,7 +1344,8 @@ static void handle_reply(struct ceph_osd_client *osdc, struct ceph_msg *msg,
struct ceph_osd_request *req;
u64 tid;
int object_len;
- int numops, payload_len, flags;
+ unsigned int numops;
+ int payload_len, flags;
s32 result;
s32 retry_attempt;
struct ceph_pg pg;
@@ -1352,7 +1355,7 @@ static void handle_reply(struct ceph_osd_client *osdc, struct ceph_msg *msg,
u32 osdmap_epoch;
int already_completed;
u32 bytes;
- int i;
+ unsigned int i;

tid = le64_to_cpu(msg->hdr.tid);
dout("handle_reply %p tid %llu\n", msg, tid);
@@ -2116,12 +2119,11 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
struct page **pages, int num_pages, int page_align)
{
struct ceph_osd_request *req;
- struct ceph_osd_req_op op;
int rc = 0;

dout("readpages on ino %llx.%llx on %llu~%llu\n", vino.ino,
vino.snap, off, *plen);
- req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1, &op,
+ req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1,
CEPH_OSD_OP_READ, CEPH_OSD_FLAG_READ, NULL, truncate_seq, truncate_size, false);
@@ -2136,7 +2138,7 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
dout("readpages final extent is %llu~%llu (%llu bytes align %d)\n",
off, *plen, *plen, page_align);

- ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
+ ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);

rc = ceph_osdc_start_request(osdc, req, false);
if (!rc)
@@ -2160,12 +2162,11 @@ int ceph_osdc_writepages(struct ceph_osd_client *osdc, struct ceph_vino vino,
struct page **pages, int num_pages)
{
struct ceph_osd_request *req;
- struct ceph_osd_req_op op;
int rc = 0;
int page_align = off & ~PAGE_MASK;

BUG_ON(vino.snap != CEPH_NOSNAP); /* snapshots aren't writeable */
- req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1, &op,
+ req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1,
CEPH_OSD_OP_WRITE, CEPH_OSD_FLAG_ONDISK | CEPH_OSD_FLAG_WRITE, snapc, truncate_seq, truncate_size,
@@ -2178,7 +2179,7 @@ int ceph_osdc_writepages(struct ceph_osd_client *osdc, struct ceph_vino vino,
false, false);
dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);

- ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);
+ ceph_osdc_build_request(req, off, snapc, CEPH_NOSNAP, mtime);

rc = ceph_osdc_start_request(osdc, req, true); if (!rc)

Signature · 1 line
1.7.9.5
Reviewed-by on v1 8/20
#16 lore raw

Re: [PATCH 08/20] libceph: keep source rather than message osd op array

Quoted 671 lines · Alex Elder

An osd request keeps a pointer to the osd operations (ops) array that it builds in its request message.

In order to allow each op in the array to have its own distinct data, we will need to keep track of each op's data, and that information does not go over the wire.

As long as we're tracking the data we might as well just track the entire (source) op definition for each of the ops. And if we're doing that, we'll have no more need to keep a pointer to the wire-encoded version.

This patch makes the array of source ops be kept with the osd request structure, and uses that instead of the version encoded in the message in places where that was previously used. The array will be embedded in the request structure, and the maximum number of ops we ever actually use is currently 2. So reduce CEPH_OSD_MAX_OP to 2 to reduce the size of the structure.

The result of doing this sort of ripples back up, and as a result various function parameters and local variables become unnecessary.

Make r_num_ops be unsigned, and move the definition of struct ceph_osd_req_op earlier to ensure it's defined where needed.

It does not yet add per-op data, that's coming soon.

This resolves:

     http://tracker.ceph.com/issues/4656
---
  drivers/block/rbd.c             |   42 ++++++++++++-----------
  fs/ceph/addr.c                  |   21 +++++-------
  fs/ceph/file.c                  |    6 ++--
  include/linux/ceph/osd_client.h |   70
+++++++++++++++++++--------------------
  net/ceph/debugfs.c              |    4 +--
  net/ceph/osd_client.c           |   53 ++++++++++++++---------------
  6 files changed, 97 insertions(+), 99 deletions(-)
drivers/block/rbd.c +1 -1
index fc41675..e6e2191 100644
@@ -1285,7 +1285,7 @@ static void rbd_osd_req_callback(struct
 ceph_osd_request *osd_req,
  	 */
  	obj_request->xferred = osd_req->r_reply_op_len[0];
  	rbd_assert(obj_request->xferred < (u64) UINT_MAX);
-	opcode = osd_req->r_request_ops[0].op;
+	opcode = osd_req->r_ops[0].op;
  	switch (opcode) {
  	case CEPH_OSD_OP_READ:
  		rbd_osd_read_callback(obj_request);
drivers/block/rbd.c +1 -2
@@ -1312,8 +1312,7 @@ static void rbd_osd_req_callback(struct
 ceph_osd_request *osd_req,
  }
 
  static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
-					bool write_request,
-					struct ceph_osd_req_op *op)
+					bool write_request)
  {
  	struct rbd_img_request *img_request = obj_request->img_request;
  	struct ceph_snap_context *snapc = NULL;
drivers/block/rbd.c +1 -1
@@ -1333,7 +1332,7 @@ static void rbd_osd_req_format_op(struct
 rbd_obj_request *obj_request,
  	}
 
  	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
-			1, op, snapc, snap_id, mtime);
+			snapc, snap_id, mtime);
  }
 

static struct ceph_osd_request *rbd_osd_req_create(

drivers/block/rbd.c +8 -6
@@ -1562,7 +1561,7 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  	while (resid) {
  		const char *object_name;
  		unsigned int clone_size;
-		struct ceph_osd_req_op op;
+		struct ceph_osd_req_op *op;
  		u64 offset;
  		u64 length;
@@ -1591,8 +1590,9 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  		if (!obj_request->osd_req)
  			goto out_partial;
 
-		osd_req_op_extent_init(&op, opcode, offset, length, 0, 0);
-		rbd_osd_req_format_op(obj_request, write_request, &op);
+		op = &obj_request->osd_req->r_ops[0];
+		osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
+		rbd_osd_req_format_op(obj_request, write_request);
 
  		/* status and version are initially zero-filled */
@@ -1693,7 +1693,7 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
  				   u64 ver, u64 notify_id)
  {
  	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op op;
+	struct ceph_osd_req_op *op;
  	struct ceph_osd_client *osdc;
  	int ret;
@@ -1707,8 +1707,9 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	osd_req_op_watch_init(&op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
-	rbd_osd_req_format_op(obj_request, false, &op);
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+	rbd_osd_req_format_op(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	obj_request->callback = rbd_obj_request_put;
drivers/block/rbd.c +1 -1
@@ -1748,7 +1749,7 @@ static int rbd_dev_header_watch_sync(struct
 rbd_device *rbd_dev, int start)
  {
  	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
  	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op op;
+	struct ceph_osd_req_op *op;
  	int ret;
 
  	rbd_assert(start ^ !!rbd_dev->watch_event);
drivers/block/rbd.c +3 -2
@@ -1772,10 +1773,11 @@ static int rbd_dev_header_watch_sync(struct
 rbd_device *rbd_dev, int start)
  	if (!obj_request->osd_req)
  		goto out_cancel;
 
-	osd_req_op_watch_init(&op, CEPH_OSD_OP_WATCH,
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
  				rbd_dev->watch_event->cookie,
  				rbd_dev->header.obj_version, start);
-	rbd_osd_req_format_op(obj_request, true, &op);
+	rbd_osd_req_format_op(obj_request, true);
 
  	if (start)
  		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
drivers/block/rbd.c +1 -1
@@ -1835,7 +1837,7 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  {
  	struct rbd_obj_request *obj_request;
  	struct ceph_osd_client *osdc;
-	struct ceph_osd_req_op op;
+	struct ceph_osd_req_op *op;
  	struct page **pages;
  	u32 page_count;

int ret;

drivers/block/rbd.c +3 -2
@@ -1865,9 +1867,10 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	osd_req_op_cls_init(&op, CEPH_OSD_OP_CALL, class_name, method_name,
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
  					outbound, outbound_size);
-	rbd_osd_req_format_op(obj_request, false, &op);
+	rbd_osd_req_format_op(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	ret = rbd_obj_request_submit(osdc, obj_request);
drivers/block/rbd.c +1 -1
@@ -2045,8 +2048,8 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  				char *buf, u64 *version)
 
  {
-	struct ceph_osd_req_op op;
  	struct rbd_obj_request *obj_request;
+	struct ceph_osd_req_op *op;
  	struct ceph_osd_client *osdc;
  	struct page **pages = NULL;

u32 page_count;

drivers/block/rbd.c +3 -2
@@ -2071,8 +2074,9 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	osd_req_op_extent_init(&op, CEPH_OSD_OP_READ, offset, length, 0, 0);
-	rbd_osd_req_format_op(obj_request, false, &op);
+	op = &obj_request->osd_req->r_ops[0];
+	osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+	rbd_osd_req_format_op(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	ret = rbd_obj_request_submit(osdc, obj_request);
fs/ceph/addr.c +0 -1
index c46b9d3..12bbba7 100644
@@ -288,7 +288,6 @@ static int start_read(struct inode *inode, struct
 list_head *page_list, int max)
  	struct page *page = list_entry(page_list->prev, struct page, lru);
  	struct ceph_vino vino;
  	struct ceph_osd_request *req;
-	struct ceph_osd_req_op op;
  	u64 off;
  	u64 len;

int i;

fs/ceph/addr.c +1 -1
@@ -314,7 +313,7 @@ static int start_read(struct inode *inode, struct
 list_head *page_list, int max)
  	     off, len);
  	vino = ceph_vino(inode);
  	req = ceph_osdc_new_request(osdc, &ci->i_layout, vino, off, &len,
-				    1, &op, CEPH_OSD_OP_READ,
+				    1, CEPH_OSD_OP_READ,
  				    CEPH_OSD_FLAG_READ, NULL,
  				    ci->i_truncate_seq, ci->i_truncate_size,

false);

fs/ceph/addr.c +1 -1
@@ -349,7 +348,7 @@ static int start_read(struct inode *inode, struct
 list_head *page_list, int max)
  	req->r_callback = finish_read;
  	req->r_inode = inode;
 
-	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
+	ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);
 
  	dout("start_read %p starting %p %lld~%lld\n", inode, req, off, len);
  	ret = ceph_osdc_start_request(osdc, req, false);
fs/ceph/addr.c +1 -1
@@ -567,7 +566,7 @@ static void writepages_finish(struct
 ceph_osd_request *req,
  	struct ceph_snap_context *snapc = req->r_snapc;
  	struct address_space *mapping = inode->i_mapping;
  	int rc = req->r_result;
-	u64 bytes = le64_to_cpu(req->r_request_ops[0].extent.length);
+	u64 bytes = req->r_ops[0].extent.length;
  	struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
  	long writeback_stat;
  	unsigned issued = ceph_caps_issued(ci);
fs/ceph/addr.c +1 -2
@@ -635,8 +634,7 @@ static void writepages_finish(struct
 ceph_osd_request *req,
 
  static struct ceph_osd_request *
  ceph_writepages_osd_request(struct inode *inode, u64 offset, u64 *len,
-				struct ceph_snap_context *snapc,
-				int num_ops, struct ceph_osd_req_op *ops)
+				struct ceph_snap_context *snapc, int num_ops)
  {
  	struct ceph_fs_client *fsc;

struct ceph_inode_info *ci;

fs/ceph/addr.c +1 -1
@@ -648,7 +646,7 @@ ceph_writepages_osd_request(struct inode *inode, u64
 offset, u64 *len,
  	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
 
  	return ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
-			vino, offset, len, num_ops, ops, CEPH_OSD_OP_WRITE,
+			vino, offset, len, num_ops, CEPH_OSD_OP_WRITE,
  			CEPH_OSD_FLAG_WRITE|CEPH_OSD_FLAG_ONDISK,
  			snapc, ci->i_truncate_seq, ci->i_truncate_size, true);
  }
fs/ceph/addr.c +4 -5
@@ -738,7 +736,6 @@ retry:
  	last_snapc = snapc;
 
  	while (!done && index <= end) {
-		struct ceph_osd_req_op ops[2];
  		int num_ops = do_sync ? 2 : 1;
  		struct ceph_vino vino;
  		unsigned i;
@@ -846,7 +843,7 @@ get_more_pages:
  				len = wsize;
  				req = ceph_writepages_osd_request(inode,
  							offset, &len, snapc,
-							num_ops, ops);
+							num_ops);
 
  				if (IS_ERR(req)) {
  					rc = PTR_ERR(req);
@@ -927,11 +924,11 @@ get_more_pages:
 
  		/* Update the write op length in case we changed it */
 
-		osd_req_op_extent_update(&ops[0], len);
+		osd_req_op_extent_update(&req->r_ops[0], len);
 
  		vino = ceph_vino(inode);
-		ceph_osdc_build_request(req, offset, num_ops, ops,
-					snapc, vino.snap, &inode->i_mtime);
+		ceph_osdc_build_request(req, offset, snapc, vino.snap,
+					&inode->i_mtime);
 
  		rc = ceph_osdc_start_request(&fsc->client->osdc, req, true);
  		BUG_ON(rc);
fs/ceph/file.c +0 -1
index 2f2d0a1..a1d85a8 100644
@@ -478,7 +478,6 @@ static ssize_t ceph_sync_write(struct file *file,
 const char __user *data,
  	struct ceph_snap_context *snapc;
  	struct ceph_vino vino;
  	struct ceph_osd_request *req;
-	struct ceph_osd_req_op ops[2];
  	int num_ops = 1;
  	struct page **pages;

int num_pages;

fs/ceph/file.c +2 -3
@@ -534,7 +533,7 @@ more:
  	snapc = ci->i_snap_realm->cached_context;
  	vino = ceph_vino(inode);
  	req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout,
-				    vino, pos, &len, num_ops, ops,
+				    vino, pos, &len, num_ops,
  				    CEPH_OSD_OP_WRITE, flags, snapc,
  				    ci->i_truncate_seq, ci->i_truncate_size,
  				    false);
@@ -579,8 +578,7 @@ more:
  					false, own_pages);
 
  	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
-	ceph_osdc_build_request(req, pos, num_ops, ops,
-				snapc, vino.snap, &mtime);
+	ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);
 
  	ret = ceph_osdc_start_request(&fsc->client->osdc, req, false);
  	if (!ret) {
include/linux/ceph/osd_client.h +33 -34
index af60dac..f4c1a2a 100644
@@ -48,7 +48,7 @@ struct ceph_osd {
  };
 
 
-#define CEPH_OSD_MAX_OP 10
+#define CEPH_OSD_MAX_OP	2
 
  enum ceph_osd_data_type {
  	CEPH_OSD_DATA_TYPE_NONE,
@@ -79,6 +79,34 @@ struct ceph_osd_data {
  	};
  };
 
+struct ceph_osd_req_op {
+	u16 op;           /* CEPH_OSD_OP_* */
+	u32 payload_len;
+	union {
+		struct {
+			u64 offset, length;
+			u64 truncate_size;
+			u32 truncate_seq;
+		} extent;
+		struct {
+			const char *class_name;
+			const char *method_name;
+			const void *indata;
+			u32 indata_len;
+			__u8 class_len;
+			__u8 method_len;
+			__u8 argc;
+		} cls;
+		struct {
+			u64 cookie;
+			u64 ver;
+			u32 prot_ver;
+			u32 timeout;
+			__u8 flag;
+		} watch;
+	};
+};
+
  /* an in-flight request */
  struct ceph_osd_request {
  	u64             r_tid;              /* unique for this client */
@@ -95,10 +123,11 @@ struct ceph_osd_request {
  	struct ceph_msg  *r_request, *r_reply;
  	int               r_flags;     /* any additional flags for the osd */
  	u32               r_sent;      /* >0 if r_request is sending/sent */
-	int               r_num_ops;
 
-	/* encoded message content */
-	struct ceph_osd_op *r_request_ops;
+	/* request osd ops array  */
+	unsigned int		r_num_ops;
+	struct ceph_osd_req_op	r_ops[CEPH_OSD_MAX_OP];
+
  	/* these are updated on each send */
  	__le32           *r_request_osdmap_epoch;
  	__le32           *r_request_flags;
@@ -193,34 +222,6 @@ struct ceph_osd_client {
  	struct workqueue_struct	*notify_wq;
  };
 
-struct ceph_osd_req_op {
-	u16 op;           /* CEPH_OSD_OP_* */
-	u32 payload_len;
-	union {
-		struct {
-			u64 offset, length;
-			u64 truncate_size;
-			u32 truncate_seq;
-		} extent;
-		struct {
-			const char *class_name;
-			const char *method_name;
-			const void *indata;
-			u32 indata_len;
-			__u8 class_len;
-			__u8 method_len;
-			__u8 argc;
-		} cls;
-		struct {
-			u64 cookie;
-			u64 ver;
-			u32 prot_ver;
-			u32 timeout;
-			__u8 flag;
-		} watch;
-	};
-};
-
  extern int ceph_osdc_init(struct ceph_osd_client *osdc,
  			  struct ceph_client *client);
  extern void ceph_osdc_stop(struct ceph_osd_client *osdc);
@@ -249,8 +250,6 @@ extern struct ceph_osd_request
 *ceph_osdc_alloc_request(struct ceph_osd_client *
  					       gfp_t gfp_flags);
 
  extern void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
-				    unsigned int num_ops,
-				    struct ceph_osd_req_op *src_ops,
  				    struct ceph_snap_context *snapc,
  				    u64 snap_id,

struct timespec *mtime);

include/linux/ceph/osd_client.h +1 -2
@@ -259,8 +258,7 @@ extern struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *,
  				      struct ceph_file_layout *layout,
  				      struct ceph_vino vino,
  				      u64 offset, u64 *len,
-				      int num_ops, struct ceph_osd_req_op *ops,
-				      int opcode, int flags,
+				      int num_ops, int opcode, int flags,
  				      struct ceph_snap_context *snapc,
  				      u32 truncate_seq, u64 truncate_size,

bool use_mempool);

net/ceph/debugfs.c +2 -2
index 00d051f..83661cd 100644
@@ -123,8 +123,8 @@ static int osdc_show(struct seq_file *s, void *pp)
  	mutex_lock(&osdc->request_mutex);
  	for (p = rb_first(&osdc->requests); p; p = rb_next(p)) {
  		struct ceph_osd_request *req;
+		unsigned int i;
  		int opcode;
-		int i;
 
  		req = rb_entry(p, struct ceph_osd_request, r_node);
 
@@ -142,7 +142,7 @@ static int osdc_show(struct seq_file *s, void *pp)
  			seq_printf(s, "\t");
 
  		for (i = 0; i < req->r_num_ops; i++) {
-			opcode = le16_to_cpu(req->r_request_ops[i].op);
+			opcode = req->r_ops[i].op;
  			seq_printf(s, "\t%s", ceph_osd_op_name(opcode));
  		}
 
net/ceph/osd_client.c +3 -0
index 110cd83..ed4493d 100644
@@ -186,6 +186,9 @@ struct ceph_osd_request
 *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
  	struct ceph_msg *msg;
  	size_t msg_size;
 
+	BUILD_BUG_ON(CEPH_OSD_MAX_OP > U16_MAX);
+	BUG_ON(num_ops > CEPH_OSD_MAX_OP);
+
  	msg_size = 4 + 4 + 8 + 8 + 4+8;
  	msg_size += 2 + 4 + 8 + 4 + 4; /* oloc */

msg_size += 1 + 8 + 4 + 4; /* pg_t */

net/ceph/osd_client.c +1 -0
@@ -207,6 +210,7 @@ struct ceph_osd_request
 *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
 
  	req->r_osdc = osdc;
  	req->r_mempool = use_mempool;
+	req->r_num_ops = num_ops;
 
  	kref_init(&req->r_kref);
  	init_completion(&req->r_completion);
net/ceph/osd_client.c +11 -16
@@ -418,12 +422,14 @@ void osd_req_op_watch_init(struct ceph_osd_req_op
 *op, u16 opcode,
  EXPORT_SYMBOL(osd_req_op_watch_init);
 
  static u64 osd_req_encode_op(struct ceph_osd_request *req,
-			      struct ceph_osd_op *dst,
-			      struct ceph_osd_req_op *src)
+			      struct ceph_osd_op *dst, unsigned int which)
  {
+	struct ceph_osd_req_op *src;
  	u64 out_data_len = 0;
  	struct ceph_pagelist *pagelist;
 
+	BUG_ON(which >= req->r_num_ops);
+	src = &req->r_ops[which];
  	if (WARN_ON(!osd_req_opcode_valid(src->op))) {
  		pr_err("unrecognized osd opcode %d\n", src->op);
@@ -487,21 +493,17 @@ static u64 osd_req_encode_op(struct
 ceph_osd_request *req,
   * build new request AND message
   *
   */
-void ceph_osdc_build_request(struct ceph_osd_request *req,
-			     u64 off, unsigned int num_ops,
-			     struct ceph_osd_req_op *src_ops,
-			     struct ceph_snap_context *snapc, u64 snap_id,
-			     struct timespec *mtime)
+void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
+				struct ceph_snap_context *snapc, u64 snap_id,
+				struct timespec *mtime)
  {
  	struct ceph_msg *msg = req->r_request;
-	struct ceph_osd_req_op *src_op;
  	void *p;
  	size_t msg_size;
  	int flags = req->r_flags;
  	u64 data_len;
-	int i;
+	unsigned int i;
 
-	req->r_num_ops = num_ops;
  	req->r_snapid = snap_id;
  	req->r_snapc = ceph_get_snap_context(snapc);
@@ -541,12 +543,10 @@ void ceph_osdc_build_request(struct
 ceph_osd_request *req,
  	p += req->r_oid_len;
 
  	/* ops--can imply data */
-	ceph_encode_16(&p, num_ops);
-	src_op = src_ops;
-	req->r_request_ops = p;
+	ceph_encode_16(&p, (u16)req->r_num_ops);
  	data_len = 0;
-	for (i = 0; i < num_ops; i++, src_op++) {
-		data_len += osd_req_encode_op(req, p, src_op);
+	for (i = 0; i < req->r_num_ops; i++) {
+		data_len += osd_req_encode_op(req, p, i);
  		p += sizeof(struct ceph_osd_op);
  	}
@@ -602,7 +602,6 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  					       struct ceph_file_layout *layout,
  					       struct ceph_vino vino,
  					       u64 off, u64 *plen, int num_ops,
-					       struct ceph_osd_req_op *ops,
  					       int opcode, int flags,
  					       struct ceph_snap_context *snapc,

u32 truncate_seq,

net/ceph/osd_client.c +1 -0
@@ -610,6 +609,7 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  					       bool use_mempool)
  {
  	struct ceph_osd_request *req;
+	struct ceph_osd_req_op *op;
  	u64 objnum = 0;
  	u64 objoff = 0;
  	u64 objlen = 0;
net/ceph/osd_client.c +1 -0
@@ -623,6 +623,7 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  					GFP_NOFS);
  	if (!req)
  		return ERR_PTR(-ENOMEM);
+
  	req->r_flags = flags;
 

/* calculate max write size */

net/ceph/osd_client.c +2 -1
@@ -642,7 +643,8 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  			truncate_size = object_size;
  	}
 
-	osd_req_op_extent_init(&ops[0], opcode, objoff, objlen,
+	op = &req->r_ops[0];
+	osd_req_op_extent_init(op, opcode, objoff, objlen,
  				truncate_size, truncate_seq);
  	/*

* A second op in the ops array means the caller wants to

net/ceph/osd_client.c +3 -2
@@ -650,7 +652,7 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  	 * osd will flush data quickly.
  	 */
  	if (num_ops > 1)
-		osd_req_op_init(&ops[1], CEPH_OSD_OP_STARTSYNC);
+		osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);
 
  	req->r_file_layout = *layout;  /* keep a copy */
@@ -1342,7 +1344,8 @@ static void handle_reply(struct ceph_osd_client
 *osdc, struct ceph_msg *msg,
  	struct ceph_osd_request *req;
  	u64 tid;
  	int object_len;
-	int numops, payload_len, flags;
+	unsigned int numops;
+	int payload_len, flags;
  	s32 result;
  	s32 retry_attempt;

struct ceph_pg pg;

net/ceph/osd_client.c +1 -1
@@ -1352,7 +1355,7 @@ static void handle_reply(struct ceph_osd_client
 *osdc, struct ceph_msg *msg,
  	u32 osdmap_epoch;
  	int already_completed;
  	u32 bytes;
-	int i;
+	unsigned int i;
 
  	tid = le64_to_cpu(msg->hdr.tid);
  	dout("handle_reply %p tid %llu\n", msg, tid);
net/ceph/osd_client.c +1 -2
@@ -2116,12 +2119,11 @@ int ceph_osdc_readpages(struct ceph_osd_client
 *osdc,
  			struct page **pages, int num_pages, int page_align)
  {
  	struct ceph_osd_request *req;
-	struct ceph_osd_req_op op;
  	int rc = 0;
 
  	dout("readpages on ino %llx.%llx on %llu~%llu\n", vino.ino,
  	     vino.snap, off, *plen);
-	req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1, &op,
+	req = ceph_osdc_new_request(osdc, layout, vino, off, plen, 1,
  				    CEPH_OSD_OP_READ, CEPH_OSD_FLAG_READ,
  				    NULL, truncate_seq, truncate_size,

false);

net/ceph/osd_client.c +2 -3
@@ -2136,7 +2138,7 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
  	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
  	     off, *plen, *plen, page_align);
 
-	ceph_osdc_build_request(req, off, 1, &op, NULL, vino.snap, NULL);
+	ceph_osdc_build_request(req, off, NULL, vino.snap, NULL);
 
  	rc = ceph_osdc_start_request(osdc, req, false);
  	if (!rc)
@@ -2160,12 +2162,11 @@ int ceph_osdc_writepages(struct ceph_osd_client
 *osdc, struct ceph_vino vino,
  			 struct page **pages, int num_pages)
  {
  	struct ceph_osd_request *req;
-	struct ceph_osd_req_op op;
  	int rc = 0;
  	int page_align = off & ~PAGE_MASK;
 
  	BUG_ON(vino.snap != CEPH_NOSNAP);	/* snapshots aren't writeable */
-	req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1, &op,
+	req = ceph_osdc_new_request(osdc, layout, vino, off, &len, 1,
  				    CEPH_OSD_OP_WRITE,
  				    CEPH_OSD_FLAG_ONDISK | CEPH_OSD_FLAG_WRITE,

snapc, truncate_seq, truncate_size,

net/ceph/osd_client.c +1 -1
@@ -2178,7 +2179,7 @@ int ceph_osdc_writepages(struct ceph_osd_client
 *osdc, struct ceph_vino vino,
  				false, false);
  	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);
 
-	ceph_osdc_build_request(req, off, 1, &op, snapc, CEPH_NOSNAP, mtime);
+	ceph_osdc_build_request(req, off, snapc, CEPH_NOSNAP, mtime);
 
  	rc = ceph_osdc_start_request(osdc, req, true);

if (!rc)

author
#17 lore raw

PATCH 9/20 libceph: rename data out field in osd request op

In reply to Alex Elder

There are fields "indata" and "indata_len" defined the ceph osd request op structure. The "in" part is with from the point of view of the osd server, but is a little confusing here on the client side. Change their names to use "request" instead of "in" to indicate that it defines data provided with the request (as opposed the data returned in the response).

Rename the local variable in osd_req_encode_op() to match.

---
 include/linux/ceph/osd_client.h |    4 ++--
 net/ceph/osd_client.c           |   18 +++++++++---------
 2 files changed, 11 insertions(+), 11 deletions(-)
(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index f4c1a2a..a9c4089 100644

include/linux/ceph/osd_client.h +2 -2
@@ -91,8 +91,8 @@ struct ceph_osd_req_op {
 		struct {
 			const char *class_name;
 			const char *method_name;
-			const void *indata;
-			u32 indata_len;
+			const void *request_data;
+			u32 request_data_len;
 			__u8 class_len;
 			__u8 method_len;
 			__u8 argc;
net/ceph/osd_client.c +0 -0
index ed4493d..e74e454 100644
@@ -395,9 +395,9 @@ void osd_req_op_cls_init(struct ceph_osd_req_op *op,

u16 opcode,
op->cls.method_len = size;
payload_len += size;

- op->cls.indata = request_data;
+ op->cls.request_data = request_data;
BUG_ON(request_data_size > (size_t) U32_MAX);
- op->cls.indata_len = (u32) request_data_size;
+ op->cls.request_data_len = (u32) request_data_size;
payload_len += request_data_size;

op->cls.argc = 0; /* currently unused */
@@ -425,7 +425,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
struct ceph_osd_op *dst, unsigned int which)
{
struct ceph_osd_req_op *src;
- u64 out_data_len = 0;
+ u64 request_data_len = 0;
struct ceph_pagelist *pagelist;

BUG_ON(which >= req->r_num_ops);
@@ -442,7 +442,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
case CEPH_OSD_OP_READ:
case CEPH_OSD_OP_WRITE:
if (src->op == CEPH_OSD_OP_WRITE)
- out_data_len = src->extent.length;
+ request_data_len = src->extent.length;
dst->extent.offset = cpu_to_le64(src->extent.offset); dst->extent.length = cpu_to_le64(src->extent.length); dst->extent.truncate_size =
@@ -457,16 +457,16 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,

dst->cls.class_len = src->cls.class_len; dst->cls.method_len = src->cls.method_len;
- dst->cls.indata_len = cpu_to_le32(src->cls.indata_len);
+ dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
ceph_pagelist_append(pagelist, src->cls.class_name,
src->cls.class_len);
ceph_pagelist_append(pagelist, src->cls.method_name,
src->cls.method_len);
- ceph_pagelist_append(pagelist, src->cls.indata,
- src->cls.indata_len);
+ ceph_pagelist_append(pagelist, src->cls.request_data,
+ src->cls.request_data_len);

ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
- out_data_len = pagelist->length;
+ request_data_len = pagelist->length;
break;
case CEPH_OSD_OP_STARTSYNC:
break;
@@ -486,7 +486,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
dst->op = cpu_to_le16(src->op);
dst->payload_len = cpu_to_le32(src->payload_len);

- return out_data_len;
+ return request_data_len;
}

/*

Signature · 1 line
1.7.9.5
Reviewed-by on v1 9/20
#18 lore raw

Re: [PATCH 09/20] libceph: rename data out field in osd request op

Quoted 101 lines · Alex Elder

There are fields "indata" and "indata_len" defined the ceph osd request op structure. The "in" part is with from the point of view of the osd server, but is a little confusing here on the client side. Change their names to use "request" instead of "in" to indicate that it defines data provided with the request (as opposed the data returned in the response).

Rename the local variable in osd_req_encode_op() to match.

---
  include/linux/ceph/osd_client.h |    4 ++--
  net/ceph/osd_client.c           |   18 +++++++++---------
  2 files changed, 11 insertions(+), 11 deletions(-)
include/linux/ceph/osd_client.h +2 -2
index f4c1a2a..a9c4089 100644
@@ -91,8 +91,8 @@ struct ceph_osd_req_op {
  		struct {
  			const char *class_name;
  			const char *method_name;
-			const void *indata;
-			u32 indata_len;
+			const void *request_data;
+			u32 request_data_len;
  			__u8 class_len;
  			__u8 method_len;
  			__u8 argc;
net/ceph/osd_client.c +2 -2
index ed4493d..e74e454 100644
@@ -395,9 +395,9 @@ void osd_req_op_cls_init(struct ceph_osd_req_op *op,
 u16 opcode,
  	op->cls.method_len = size;
  	payload_len += size;
 
-	op->cls.indata = request_data;
+	op->cls.request_data = request_data;
  	BUG_ON(request_data_size > (size_t) U32_MAX);
-	op->cls.indata_len = (u32) request_data_size;
+	op->cls.request_data_len = (u32) request_data_size;
  	payload_len += request_data_size;
 

op->cls.argc = 0; /* currently unused */

net/ceph/osd_client.c +1 -1
@@ -425,7 +425,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
 *req,
  			      struct ceph_osd_op *dst, unsigned int which)
  {
  	struct ceph_osd_req_op *src;
-	u64 out_data_len = 0;
+	u64 request_data_len = 0;
  	struct ceph_pagelist *pagelist;
 
  	BUG_ON(which >= req->r_num_ops);
net/ceph/osd_client.c +1 -1
@@ -442,7 +442,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
 *req,
  	case CEPH_OSD_OP_READ:
  	case CEPH_OSD_OP_WRITE:
  		if (src->op == CEPH_OSD_OP_WRITE)
-			out_data_len = src->extent.length;
+			request_data_len = src->extent.length;
  		dst->extent.offset = cpu_to_le64(src->extent.offset);
  		dst->extent.length = cpu_to_le64(src->extent.length);

dst->extent.truncate_size =

net/ceph/osd_client.c +4 -4
@@ -457,16 +457,16 @@ static u64 osd_req_encode_op(struct
 ceph_osd_request *req,
 
  		dst->cls.class_len = src->cls.class_len;
  		dst->cls.method_len = src->cls.method_len;
-		dst->cls.indata_len = cpu_to_le32(src->cls.indata_len);
+		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
  		ceph_pagelist_append(pagelist, src->cls.class_name,
  				     src->cls.class_len);
  		ceph_pagelist_append(pagelist, src->cls.method_name,
  				     src->cls.method_len);
-		ceph_pagelist_append(pagelist, src->cls.indata,
-				     src->cls.indata_len);
+		ceph_pagelist_append(pagelist, src->cls.request_data,
+				     src->cls.request_data_len);
 
  		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
-		out_data_len = pagelist->length;
+		request_data_len = pagelist->length;
  		break;
  	case CEPH_OSD_OP_STARTSYNC:

break;

net/ceph/osd_client.c +1 -1
@@ -486,7 +486,7 @@ static u64 osd_req_encode_op(struct ceph_osd_request
 *req,
  	dst->op = cpu_to_le16(src->op);
  	dst->payload_len = cpu_to_le32(src->payload_len);
 
-	return out_data_len;
+	return request_data_len;
  }
 

/*

author
#19 lore raw

PATCH 10/20 libceph: add data pointers in osd op structures

In reply to Alex Elder

An extent type osd operation currently implies that there will be corresponding data supplied in the data portion of the request (for write) or response (for read) message. Similarly, an osd class method operation implies a data item will be supplied to receive the response data from the operation.

Add a ceph_osd_data pointer to each of those structures, and assign it to point to eithre the incoming or the outgoing data structure in the osd message. The data is not always available when an op is initially set up, so add two new functions to allow setting them after the op has been initialized.

Begin to make use of the data item pointer available in the osd operation rather than the request data in or out structure in places where it's convenient. Add some assertions to verify pointers are always set the way they're expected to be.

This is a sort of stepping stone toward really moving the data into the osd request ops, to allow for some validation before making that jump.

This is the first in a series of patches that resolve:

    http://tracker.ceph.com/issues/4657
---
 drivers/block/rbd.c             |   24 ++++++++++++++++++++----
 fs/ceph/addr.c                  |    8 +++++---
 fs/ceph/file.c                  |    5 +++--
 include/linux/ceph/osd_client.h |    6 ++++++
 net/ceph/osd_client.c           |   26 +++++++++++++++++++++++++-
 5 files changed, 59 insertions(+), 10 deletions(-)
drivers/block/rbd.c +0 -0
index e6e2191..59f9db1 100644
@@ -1315,23 +1315,39 @@ static void rbd_osd_req_format_op(struct

rbd_obj_request *obj_request,
bool write_request)
{
struct rbd_img_request *img_request = obj_request->img_request;
+ struct ceph_osd_request *osd_req = obj_request->osd_req;
+ struct ceph_osd_data *osd_data = NULL;
struct ceph_snap_context *snapc = NULL;
u64 snap_id = CEPH_NOSNAP;
struct timespec *mtime = NULL;
struct timespec now;

- rbd_assert(obj_request->osd_req != NULL);
+ rbd_assert(osd_req != NULL);

if (write_request) {
+ osd_data = &osd_req->r_data_out;
now = CURRENT_TIME;
mtime = &now;
if (img_request)
snapc = img_request->snapc;
- } else if (img_request) {
- snap_id = img_request->snap_id;
+ } else {
+ osd_data = &osd_req->r_data_in;
+ if (img_request)
+ snap_id = img_request->snap_id;
}
+ if (obj_request->type != OBJ_REQUEST_NODATA) {
+ struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];

- ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
+ /*
+ * If it has data, it's either a object class method
+ * call (cls) or it's an extent operation.
+ */
+ if (op->op == CEPH_OSD_OP_CALL)
+ osd_req_op_cls_response_data(op, osd_data);
+ else
+ osd_req_op_extent_osd_data(op, osd_data);
+ }
+ ceph_osdc_build_request(osd_req, obj_request->offset,
snapc, snap_id, mtime);
}

fs/ceph/addr.c +0 -0
index 12bbba7..3caadb0 100644
@@ -343,7 +343,8 @@ static int start_read(struct inode *inode, struct

list_head *page_list, int max)
}
pages[i] = page;
}
- ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
+ BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
+ ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
false, false);
req->r_callback = finish_read;
req->r_inode = inode;
@@ -916,8 +917,9 @@ get_more_pages:
dout("writepages got %d pages at %llu~%llu\n",
locked_pages, offset, len);

- ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
- !!pool, false);
+ BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
+ ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
+ len, 0, !!pool, false);

pages = NULL; /* request message now owns the pages array */
pool = NULL;

fs/ceph/file.c +3 -2
index a1d85a8..66d0938 100644
@@ -574,8 +574,9 @@ more:
 			own_pages = true;
 		}
 	}
-	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
-					false, own_pages);
+	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
+	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
+					page_align, false, own_pages);
 
 	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
 	ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);
(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index a9c4089..ae51935 100644

include/linux/ceph/osd_client.h +2 -0
@@ -87,12 +87,14 @@ struct ceph_osd_req_op {
 			u64 offset, length;
 			u64 truncate_size;
 			u32 truncate_seq;
+			struct ceph_osd_data *osd_data;
 		} extent;
 		struct {
 			const char *class_name;
 			const char *method_name;
 			const void *request_data;
 			u32 request_data_len;
+			struct ceph_osd_data *response_data;
 			__u8 class_len;
 			__u8 method_len;
 			__u8 argc;
@@ -236,10 +238,14 @@ extern void osd_req_op_extent_init(struct

ceph_osd_req_op *op, u16 opcode,
u64 offset, u64 length, u64 truncate_size, u32 truncate_seq);
extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64
length);
+extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+ struct ceph_osd_data *osd_data);
extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
const char *class, const char *method, const void *request_data, size_t request_data_size);
+extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+ struct ceph_osd_data *response_data);
extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
u64 cookie, u64 version, int flag);

net/ceph/osd_client.c +0 -0
index e74e454..2a14187 100644
@@ -372,6 +372,13 @@ void osd_req_op_extent_update(struct

ceph_osd_req_op *op, u64 length)
}
EXPORT_SYMBOL(osd_req_op_extent_update);

+void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+ struct ceph_osd_data *osd_data)
+{
+ op->extent.osd_data = osd_data;
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data);
+
void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
const char *class, const char *method, const void *request_data, size_t request_data_size)
@@ -406,6 +413,13 @@ void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
}
EXPORT_SYMBOL(osd_req_op_cls_init);

+void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+ struct ceph_osd_data *response_data)
+{
+ op->cls.response_data = response_data;
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data);
+
void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
u64 cookie, u64 version, int flag)
{
@@ -449,6 +463,10 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
cpu_to_le64(src->extent.truncate_size);
dst->extent.truncate_seq =
cpu_to_le32(src->extent.truncate_seq);
+ if (src->op == CEPH_OSD_OP_WRITE)
+ WARN_ON(src->extent.osd_data != &req->r_data_out);
+ else
+ WARN_ON(src->extent.osd_data != &req->r_data_in);
break;
case CEPH_OSD_OP_CALL:
pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
@@ -464,8 +482,9 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
src->cls.method_len);
ceph_pagelist_append(pagelist, src->cls.request_data,
src->cls.request_data_len);
-
ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
+
+ WARN_ON(src->cls.response_data != &req->r_data_in);
request_data_len = pagelist->length;
break;
case CEPH_OSD_OP_STARTSYNC:
@@ -609,6 +628,7 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
bool use_mempool)
{
struct ceph_osd_request *req;
+ struct ceph_osd_data *osd_data;
struct ceph_osd_req_op *op;
u64 objnum = 0;
u64 objoff = 0;
@@ -623,6 +643,8 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
GFP_NOFS);
if (!req)
return ERR_PTR(-ENOMEM);
+ osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
+ : &req->r_data_in;

req->r_flags = flags;

@@ -646,6 +668,8 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
op = &req->r_ops[0];
osd_req_op_extent_init(op, opcode, objoff, objlen,
truncate_size, truncate_seq);
+ osd_req_op_extent_osd_data(op, osd_data);
+
/*
* A second op in the ops array means the caller wants to
* also issue a include a 'startsync' command so that the

Signature · 1 line
1.7.9.5
Reviewed-by on v1 10/20
#20 lore raw

Re: [PATCH 10/20] libceph: add data pointers in osd op structures

Quoted 250 lines · Alex Elder

An extent type osd operation currently implies that there will be corresponding data supplied in the data portion of the request (for write) or response (for read) message. Similarly, an osd class method operation implies a data item will be supplied to receive the response data from the operation.

Add a ceph_osd_data pointer to each of those structures, and assign it to point to eithre the incoming or the outgoing data structure in the osd message. The data is not always available when an op is initially set up, so add two new functions to allow setting them after the op has been initialized.

Begin to make use of the data item pointer available in the osd operation rather than the request data in or out structure in places where it's convenient. Add some assertions to verify pointers are always set the way they're expected to be.

This is a sort of stepping stone toward really moving the data into the osd request ops, to allow for some validation before making that jump.

This is the first in a series of patches that resolve:

     http://tracker.ceph.com/issues/4657
---
  drivers/block/rbd.c             |   24 ++++++++++++++++++++----
  fs/ceph/addr.c                  |    8 +++++---
  fs/ceph/file.c                  |    5 +++--
  include/linux/ceph/osd_client.h |    6 ++++++
  net/ceph/osd_client.c           |   26 +++++++++++++++++++++++++-
  5 files changed, 59 insertions(+), 10 deletions(-)
drivers/block/rbd.c +20 -4
index e6e2191..59f9db1 100644
@@ -1315,23 +1315,39 @@ static void rbd_osd_req_format_op(struct
 rbd_obj_request *obj_request,
  					bool write_request)
  {
  	struct rbd_img_request *img_request = obj_request->img_request;
+	struct ceph_osd_request *osd_req = obj_request->osd_req;
+	struct ceph_osd_data *osd_data = NULL;
  	struct ceph_snap_context *snapc = NULL;
  	u64 snap_id = CEPH_NOSNAP;
  	struct timespec *mtime = NULL;
  	struct timespec now;
 
-	rbd_assert(obj_request->osd_req != NULL);
+	rbd_assert(osd_req != NULL);
 
  	if (write_request) {
+		osd_data = &osd_req->r_data_out;
  		now = CURRENT_TIME;
  		mtime = &now;
  		if (img_request)
  			snapc = img_request->snapc;
-	} else if (img_request) {
-		snap_id = img_request->snap_id;
+	} else {
+		osd_data = &osd_req->r_data_in;
+		if (img_request)
+			snap_id = img_request->snap_id;
  	}
+	if (obj_request->type != OBJ_REQUEST_NODATA) {
+		struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];
 
-	ceph_osdc_build_request(obj_request->osd_req, obj_request->offset,
+		/*
+		 * If it has data, it's either a object class method
+		 * call (cls) or it's an extent operation.
+		 */
+		if (op->op == CEPH_OSD_OP_CALL)
+			osd_req_op_cls_response_data(op, osd_data);
+		else
+			osd_req_op_extent_osd_data(op, osd_data);
+	}
+	ceph_osdc_build_request(osd_req, obj_request->offset,
  			snapc, snap_id, mtime);
  }
fs/ceph/addr.c +2 -1
index 12bbba7..3caadb0 100644
@@ -343,7 +343,8 @@ static int start_read(struct inode *inode, struct
 list_head *page_list, int max)
  		}
  		pages[i] = page;
  	}
-	ceph_osd_data_pages_init(&req->r_data_in, pages, len, 0,
+	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
+	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
  					false, false);
  	req->r_callback = finish_read;
  	req->r_inode = inode;
fs/ceph/addr.c +3 -2
@@ -916,8 +917,9 @@ get_more_pages:
  		dout("writepages got %d pages at %llu~%llu\n",
  		     locked_pages, offset, len);
 
-		ceph_osd_data_pages_init(&req->r_data_out, pages, len, 0,
-						!!pool, false);
+		BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
+		ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
+						len, 0, !!pool, false);
 
  		pages = NULL;	/* request message now owns the pages array */
  		pool = NULL;
fs/ceph/file.c +3 -2
index a1d85a8..66d0938 100644
@@ -574,8 +574,9 @@ more:
  			own_pages = true;
  		}
  	}
-	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
-					false, own_pages);
+	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
+	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
+					page_align, false, own_pages);
 
  	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
  	ceph_osdc_build_request(req, pos, snapc, vino.snap, &mtime);
include/linux/ceph/osd_client.h +6 -0
index a9c4089..ae51935 100644
@@ -87,12 +87,14 @@ struct ceph_osd_req_op {
  			u64 offset, length;
  			u64 truncate_size;
  			u32 truncate_seq;
+			struct ceph_osd_data *osd_data;
  		} extent;
  		struct {
  			const char *class_name;
  			const char *method_name;
  			const void *request_data;
  			u32 request_data_len;
+			struct ceph_osd_data *response_data;
  			__u8 class_len;
  			__u8 method_len;
  			__u8 argc;
@@ -236,10 +238,14 @@ extern void osd_req_op_extent_init(struct
 ceph_osd_req_op *op, u16 opcode,
  					u64 offset, u64 length,
  					u64 truncate_size, u32 truncate_seq);
  extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64
 length);
+extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+					struct ceph_osd_data *osd_data);
  extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
  					const char *class, const char *method,
  					const void *request_data,
  					size_t request_data_size);
+extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+					struct ceph_osd_data *response_data);
  extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,

u64 cookie, u64 version, int flag);

net/ceph/osd_client.c +7 -0
index e74e454..2a14187 100644
@@ -372,6 +372,13 @@ void osd_req_op_extent_update(struct
 ceph_osd_req_op *op, u64 length)
  }
  EXPORT_SYMBOL(osd_req_op_extent_update);
 
+void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+				struct ceph_osd_data *osd_data)
+{
+	op->extent.osd_data = osd_data;
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data);
+
  void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
  			const char *class, const char *method,

const void *request_data, size_t request_data_size)

net/ceph/osd_client.c +7 -0
@@ -406,6 +413,13 @@ void osd_req_op_cls_init(struct ceph_osd_req_op
 *op, u16 opcode,
  }
  EXPORT_SYMBOL(osd_req_op_cls_init);
 
+void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+				struct ceph_osd_data *response_data)
+{
+	op->cls.response_data = response_data;
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data);
+
  void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
  				u64 cookie, u64 version, int flag)
  {
net/ceph/osd_client.c +4 -0
@@ -449,6 +463,10 @@ static u64 osd_req_encode_op(struct
 ceph_osd_request *req,
  			cpu_to_le64(src->extent.truncate_size);
  		dst->extent.truncate_seq =
  			cpu_to_le32(src->extent.truncate_seq);
+		if (src->op == CEPH_OSD_OP_WRITE)
+			WARN_ON(src->extent.osd_data != &req->r_data_out);
+		else
+			WARN_ON(src->extent.osd_data != &req->r_data_in);
  		break;
  	case CEPH_OSD_OP_CALL:
  		pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
net/ceph/osd_client.c +2 -1
@@ -464,8 +482,9 @@ static u64 osd_req_encode_op(struct ceph_osd_request
 *req,
  				     src->cls.method_len);
  		ceph_pagelist_append(pagelist, src->cls.request_data,
  				     src->cls.request_data_len);
-
  		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
+
+		WARN_ON(src->cls.response_data != &req->r_data_in);
  		request_data_len = pagelist->length;
  		break;

case CEPH_OSD_OP_STARTSYNC:

net/ceph/osd_client.c +1 -0
@@ -609,6 +628,7 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  					       bool use_mempool)
  {
  	struct ceph_osd_request *req;
+	struct ceph_osd_data *osd_data;
  	struct ceph_osd_req_op *op;
  	u64 objnum = 0;
  	u64 objoff = 0;
net/ceph/osd_client.c +4 -0
@@ -623,6 +643,8 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  					GFP_NOFS);
  	if (!req)
  		return ERR_PTR(-ENOMEM);
+	osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
+					       : &req->r_data_in;
 
  	req->r_flags = flags;
@@ -646,6 +668,8 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  	op = &req->r_ops[0];
  	osd_req_op_extent_init(op, opcode, objoff, objlen,
  				truncate_size, truncate_seq);
+	osd_req_op_extent_osd_data(op, osd_data);
+
  	/*
  	 * A second op in the ops array means the caller wants to

* also issue a include a 'startsync' command so that the

author
#21 lore raw

PATCH 11/20 libceph: specify osd op by index in request

In reply to Alex Elder

An osd request now holds all of its source op structures, and every place that initializes one of these is in fact initializing one of the entries in the the osd request's array.

So rather than supplying the address of the op to initialize, have caller specify the osd request and an indication of which op it would like to initialize. This better hides the details the op structure (and faciltates moving the data pointers they use).

Since osd_req_op_init() is a common routine, and it's not used outside the osd client code, give it static scope. Also make it return the address of the specified op (so all the other init routines don't have to repeat that code).

---
 drivers/block/rbd.c             |   35 +++++++++------------
 fs/ceph/addr.c                  |    2 +-
 include/linux/ceph/osd_client.h |   19 +++++++-----
 net/ceph/osd_client.c           |   64
++++++++++++++++++++++++---------------
 4 files changed, 67 insertions(+), 53 deletions(-)
drivers/block/rbd.c +0 -0
index 59f9db1..7a62327 100644
@@ -1336,16 +1336,17 @@ static void rbd_osd_req_format_op(struct

rbd_obj_request *obj_request,
snap_id = img_request->snap_id;
}
if (obj_request->type != OBJ_REQUEST_NODATA) {
- struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];
-
/*
* If it has data, it's either a object class method
* call (cls) or it's an extent operation. */
- if (op->op == CEPH_OSD_OP_CALL)
- osd_req_op_cls_response_data(op, osd_data);
+ /* XXX This use of the ops array goes away in the next patch */
+ if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
+ osd_req_op_cls_response_data(obj_request->osd_req, 0,
+ osd_data);
else
- osd_req_op_extent_osd_data(op, osd_data);
+ osd_req_op_extent_osd_data(obj_request->osd_req, 0,
+ osd_data);
}
ceph_osdc_build_request(osd_req, obj_request->offset,
snapc, snap_id, mtime);
@@ -1577,7 +1578,6 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
while (resid) {
const char *object_name;
unsigned int clone_size;
- struct ceph_osd_req_op *op;
u64 offset;
u64 length;

@@ -1606,8 +1606,8 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
if (!obj_request->osd_req)
goto out_partial;

- op = &obj_request->osd_req->r_ops[0];
- osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
+ osd_req_op_extent_init(obj_request->osd_req, 0,
+ opcode, offset, length, 0, 0);
rbd_osd_req_format_op(obj_request, write_request);

/* status and version are initially zero-filled */
@@ -1709,7 +1709,6 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,
u64 ver, u64 notify_id)
{
struct rbd_obj_request *obj_request;
- struct ceph_osd_req_op *op;
struct ceph_osd_client *osdc;
int ret;

@@ -1723,8 +1722,8 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- op = &obj_request->osd_req->r_ops[0];
- osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+ osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
+ notify_id, ver, 0);
rbd_osd_req_format_op(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc;
@@ -1765,7 +1764,6 @@ static int rbd_dev_header_watch_sync(struct rbd_device *rbd_dev, int start)
{
struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc; struct rbd_obj_request *obj_request;
- struct ceph_osd_req_op *op;
int ret;

rbd_assert(start ^ !!rbd_dev->watch_event);
@@ -1789,8 +1787,7 @@ static int rbd_dev_header_watch_sync(struct rbd_device *rbd_dev, int start)
if (!obj_request->osd_req)
goto out_cancel;

- op = &obj_request->osd_req->r_ops[0];
- osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
+ osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
rbd_dev->watch_event->cookie, rbd_dev->header.obj_version, start);
rbd_osd_req_format_op(obj_request, true);
@@ -1853,7 +1850,6 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
{
struct rbd_obj_request *obj_request;
struct ceph_osd_client *osdc;
- struct ceph_osd_req_op *op;
struct page **pages;
u32 page_count;
int ret;
@@ -1883,8 +1879,8 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- op = &obj_request->osd_req->r_ops[0];
- osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
+ osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
+ class_name, method_name,
outbound, outbound_size);
rbd_osd_req_format_op(obj_request, false);

@@ -2065,7 +2061,6 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,

{
struct rbd_obj_request *obj_request;
- struct ceph_osd_req_op *op;
struct ceph_osd_client *osdc;
struct page **pages = NULL;
u32 page_count;
@@ -2090,8 +2085,8 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- op = &obj_request->osd_req->r_ops[0];
- osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+ osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
+ offset, length, 0, 0);
rbd_osd_req_format_op(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc;

fs/ceph/addr.c +1 -1
index 3caadb0..dd5d263 100644
@@ -926,7 +926,7 @@ get_more_pages:
 
 		/* Update the write op length in case we changed it */
 
-		osd_req_op_extent_update(&req->r_ops[0], len);
+		osd_req_op_extent_update(req, 0, len);
 
 		vino = ceph_vino(inode);
 		ceph_osdc_build_request(req, offset, snapc, vino.snap,
(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index ae51935..144d57c 100644

include/linux/ceph/osd_client.h +0 -0
@@ -233,20 +233,25 @@ extern void ceph_osdc_handle_reply(struct

ceph_osd_client *osdc,
extern void ceph_osdc_handle_map(struct ceph_osd_client *osdc,
struct ceph_msg *msg);

-extern void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode); -extern void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode, +extern void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
+ unsigned int which, u16 opcode,
u64 offset, u64 length, u64 truncate_size, u32 truncate_seq);
-extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64 length);
-extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op, +extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
+ unsigned int which, u64 length);
+extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+ unsigned int which,
struct ceph_osd_data *osd_data);
-extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode, +extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
+ unsigned int which, u16 opcode,
const char *class, const char *method, const void *request_data, size_t request_data_size);
-extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op, +extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+ unsigned int which,
struct ceph_osd_data *response_data);
-extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode, +extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
+ unsigned int which, u16 opcode,
u64 cookie, u64 version, int flag);

extern struct ceph_osd_request *ceph_osdc_alloc_request(struct
ceph_osd_client *osdc,

net/ceph/osd_client.c +12 -5
index 2a14187..e698ccf 100644
@@ -329,25 +329,32 @@ static bool osd_req_opcode_valid(u16 opcode)
  * other information associated with them.  It also serves as a
  * common init routine for all the other init functions, below.
  */
-void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode)
+static struct ceph_osd_req_op *
+osd_req_op_init(struct ceph_osd_request *osd_req, unsigned int which,
+				u16 opcode)
 {
+	struct ceph_osd_req_op *op;
+
+	BUG_ON(which >= osd_req->r_num_ops);
 	BUG_ON(!osd_req_opcode_valid(opcode));
 
+	op = &osd_req->r_ops[which];
 	memset(op, 0, sizeof (*op));
-
 	op->op = opcode;
+
+	return op;
 }
 
-void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode,
+void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
+				unsigned int which, u16 opcode,
 				u64 offset, u64 length,
 				u64 truncate_size, u32 truncate_seq)
 {
+	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
 	size_t payload_len = 0;
 
 	BUG_ON(opcode != CEPH_OSD_OP_READ && opcode != CEPH_OSD_OP_WRITE);
 
-	osd_req_op_init(op, opcode);
-
 	op->extent.offset = offset;
 	op->extent.length = length;
 	op->extent.truncate_size = truncate_size;
@@ -359,9 +366,15 @@ void osd_req_op_extent_init(struct ceph_osd_req_op

*op, u16 opcode,
}
EXPORT_SYMBOL(osd_req_op_extent_init);

-void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64 length) +void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
+ unsigned int which, u64 length)
{
- u64 previous = op->extent.length;
+ struct ceph_osd_req_op *op;
+ u64 previous;
+
+ BUG_ON(which >= osd_req->r_num_ops);
+ op = &osd_req->r_ops[which];
+ previous = op->extent.length;

if (length == previous)
return; /* Nothing to do */
@@ -372,24 +385,25 @@ void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64 length)
}
EXPORT_SYMBOL(osd_req_op_extent_update);

-void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op, +void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+ unsigned int which,
struct ceph_osd_data *osd_data)
{
- op->extent.osd_data = osd_data;
+ BUG_ON(which >= osd_req->r_num_ops);
+ osd_req->r_ops[which].extent.osd_data = osd_data;
}
EXPORT_SYMBOL(osd_req_op_extent_osd_data);

-void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
- const char *class, const char *method,
+void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int which,
+ u16 opcode, const char *class, const char *method,
const void *request_data, size_t request_data_size)
{
+ struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
size_t payload_len = 0;
size_t size;

BUG_ON(opcode != CEPH_OSD_OP_CALL);

- osd_req_op_init(op, opcode);
-
op->cls.class_name = class;
size = strlen(class);
BUG_ON(size > (size_t) U8_MAX);
@@ -412,26 +426,28 @@ void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
op->payload_len = payload_len;
}
EXPORT_SYMBOL(osd_req_op_cls_init);
-
-void osd_req_op_cls_response_data(struct ceph_osd_req_op *op, +void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+ unsigned int which,
struct ceph_osd_data *response_data)
{
- op->cls.response_data = response_data;
+ BUG_ON(which >= osd_req->r_num_ops);
+ osd_req->r_ops[which].cls.response_data = response_data;
}
EXPORT_SYMBOL(osd_req_op_cls_response_data);

-void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode, +void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
+ unsigned int which, u16 opcode,
u64 cookie, u64 version, int flag)
{
- BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);
+ struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);

- osd_req_op_init(op, opcode);
+ BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);

op->watch.cookie = cookie;
/* op->watch.ver = version; */ /* XXX 3847 */
op->watch.ver = cpu_to_le64(version);
if (opcode == CEPH_OSD_OP_WATCH && flag)
- op->watch.flag = (u8) 1;
+ op->watch.flag = (u8)1;
}
EXPORT_SYMBOL(osd_req_op_watch_init);

@@ -629,7 +645,6 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
{
struct ceph_osd_request *req;
struct ceph_osd_data *osd_data;
- struct ceph_osd_req_op *op;
u64 objnum = 0;
u64 objoff = 0;
u64 objlen = 0;
@@ -665,10 +680,9 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
truncate_size = object_size;
}

- op = &req->r_ops[0];
- osd_req_op_extent_init(op, opcode, objoff, objlen,
+ osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
truncate_size, truncate_seq);
- osd_req_op_extent_osd_data(op, osd_data);
+ osd_req_op_extent_osd_data(req, 0, osd_data);

/*
* A second op in the ops array means the caller wants to
@@ -676,7 +690,7 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
* osd will flush data quickly.
*/
if (num_ops > 1)
- osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);
+ osd_req_op_init(req, 1, CEPH_OSD_OP_STARTSYNC);

req->r_file_layout = *layout; /* keep a copy */

Signature · 1 line
1.7.9.5
Reviewed-by on v1 11/20
#22 lore raw

Re: [PATCH 11/20] libceph: specify osd op by index in request

Quoted 374 lines · Alex Elder

An osd request now holds all of its source op structures, and every place that initializes one of these is in fact initializing one of the entries in the the osd request's array.

So rather than supplying the address of the op to initialize, have caller specify the osd request and an indication of which op it would like to initialize. This better hides the details the op structure (and faciltates moving the data pointers they use).

Since osd_req_op_init() is a common routine, and it's not used outside the osd client code, give it static scope. Also make it return the address of the specified op (so all the other init routines don't have to repeat that code).

---
  drivers/block/rbd.c             |   35 +++++++++------------
  fs/ceph/addr.c                  |    2 +-
  include/linux/ceph/osd_client.h |   19 +++++++-----
  net/ceph/osd_client.c           |   64
++++++++++++++++++++++++---------------
  4 files changed, 67 insertions(+), 53 deletions(-)
drivers/block/rbd.c +6 -5
index 59f9db1..7a62327 100644
@@ -1336,16 +1336,17 @@ static void rbd_osd_req_format_op(struct
 rbd_obj_request *obj_request,
  			snap_id = img_request->snap_id;
  	}
  	if (obj_request->type != OBJ_REQUEST_NODATA) {
-		struct ceph_osd_req_op *op = &obj_request->osd_req->r_ops[0];
-
  		/*
  		 * If it has data, it's either a object class method
  		 * call (cls) or it's an extent operation.
  		 */
-		if (op->op == CEPH_OSD_OP_CALL)
-			osd_req_op_cls_response_data(op, osd_data);
+		/* XXX This use of the ops array goes away in the next patch */
+		if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
+			osd_req_op_cls_response_data(obj_request->osd_req, 0,
+						osd_data);
  		else
-			osd_req_op_extent_osd_data(op, osd_data);
+			osd_req_op_extent_osd_data(obj_request->osd_req, 0,
+						osd_data);
  	}
  	ceph_osdc_build_request(osd_req, obj_request->offset,

snapc, snap_id, mtime);

drivers/block/rbd.c +2 -3
@@ -1577,7 +1578,6 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  	while (resid) {
  		const char *object_name;
  		unsigned int clone_size;
-		struct ceph_osd_req_op *op;
  		u64 offset;
  		u64 length;
@@ -1606,8 +1606,8 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  		if (!obj_request->osd_req)
  			goto out_partial;
 
-		op = &obj_request->osd_req->r_ops[0];
-		osd_req_op_extent_init(op, opcode, offset, length, 0, 0);
+		osd_req_op_extent_init(obj_request->osd_req, 0,
+					opcode, offset, length, 0, 0);
  		rbd_osd_req_format_op(obj_request, write_request);
 

/* status and version are initially zero-filled */

drivers/block/rbd.c +2 -3
@@ -1709,7 +1709,6 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
  				   u64 ver, u64 notify_id)
  {
  	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op *op;
  	struct ceph_osd_client *osdc;
  	int ret;
@@ -1723,8 +1722,8 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_watch_init(op, CEPH_OSD_OP_NOTIFY_ACK, notify_id, ver, 0);
+	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
+					notify_id, ver, 0);
  	rbd_osd_req_format_op(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
drivers/block/rbd.c +0 -1
@@ -1765,7 +1764,6 @@ static int rbd_dev_header_watch_sync(struct
 rbd_device *rbd_dev, int start)
  {
  	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
  	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op *op;
  	int ret;
 
  	rbd_assert(start ^ !!rbd_dev->watch_event);
drivers/block/rbd.c +1 -2
@@ -1789,8 +1787,7 @@ static int rbd_dev_header_watch_sync(struct
 rbd_device *rbd_dev, int start)
  	if (!obj_request->osd_req)
  		goto out_cancel;
 
-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_watch_init(op, CEPH_OSD_OP_WATCH,
+	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
  				rbd_dev->watch_event->cookie,
  				rbd_dev->header.obj_version, start);
  	rbd_osd_req_format_op(obj_request, true);
drivers/block/rbd.c +0 -1
@@ -1853,7 +1850,6 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  {
  	struct rbd_obj_request *obj_request;
  	struct ceph_osd_client *osdc;
-	struct ceph_osd_req_op *op;
  	struct page **pages;
  	u32 page_count;

int ret;

drivers/block/rbd.c +2 -3
@@ -1883,8 +1879,8 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_cls_init(op, CEPH_OSD_OP_CALL, class_name, method_name,
+	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
+					class_name, method_name,
  					outbound, outbound_size);
  	rbd_osd_req_format_op(obj_request, false);
@@ -2065,7 +2061,6 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
 
  {
  	struct rbd_obj_request *obj_request;
-	struct ceph_osd_req_op *op;
  	struct ceph_osd_client *osdc;
  	struct page **pages = NULL;

u32 page_count;

drivers/block/rbd.c +2 -2
@@ -2090,8 +2085,8 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	op = &obj_request->osd_req->r_ops[0];
-	osd_req_op_extent_init(op, CEPH_OSD_OP_READ, offset, length, 0, 0);
+	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
+					offset, length, 0, 0);
  	rbd_osd_req_format_op(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
fs/ceph/addr.c +1 -1
index 3caadb0..dd5d263 100644
@@ -926,7 +926,7 @@ get_more_pages:
 
  		/* Update the write op length in case we changed it */
 
-		osd_req_op_extent_update(&req->r_ops[0], len);
+		osd_req_op_extent_update(req, 0, len);
 
  		vino = ceph_vino(inode);
  		ceph_osdc_build_request(req, offset, snapc, vino.snap,
include/linux/ceph/osd_client.h +12 -7
index ae51935..144d57c 100644
@@ -233,20 +233,25 @@ extern void ceph_osdc_handle_reply(struct
 ceph_osd_client *osdc,
  extern void ceph_osdc_handle_map(struct ceph_osd_client *osdc,
  				 struct ceph_msg *msg);
 
-extern void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode);
-extern void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode,
+extern void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
+					unsigned int which, u16 opcode,
  					u64 offset, u64 length,
  					u64 truncate_size, u32 truncate_seq);
-extern void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64
 length);
-extern void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
+					unsigned int which, u64 length);
+extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+					unsigned int which,
  					struct ceph_osd_data *osd_data);
-extern void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
+extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
+					unsigned int which, u16 opcode,
  					const char *class, const char *method,
  					const void *request_data,
  					size_t request_data_size);
-extern void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+					unsigned int which,
  					struct ceph_osd_data *response_data);
-extern void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
+extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
+					unsigned int which, u16 opcode,
  					u64 cookie, u64 version, int flag);

extern struct ceph_osd_request *ceph_osdc_alloc_request(struct
ceph_osd_client *osdc,

net/ceph/osd_client.c +20 -7
index 2a14187..e698ccf 100644
@@ -329,25 +329,32 @@ static bool osd_req_opcode_valid(u16 opcode)
   * other information associated with them.  It also serves as a
   * common init routine for all the other init functions, below.
   */
-void osd_req_op_init(struct ceph_osd_req_op *op, u16 opcode)
+static struct ceph_osd_req_op *
+osd_req_op_init(struct ceph_osd_request *osd_req, unsigned int which,
+				u16 opcode)
  {
+	struct ceph_osd_req_op *op;
+
+	BUG_ON(which >= osd_req->r_num_ops);
  	BUG_ON(!osd_req_opcode_valid(opcode));
 
+	op = &osd_req->r_ops[which];
  	memset(op, 0, sizeof (*op));
-
  	op->op = opcode;
+
+	return op;
  }
 
-void osd_req_op_extent_init(struct ceph_osd_req_op *op, u16 opcode,
+void osd_req_op_extent_init(struct ceph_osd_request *osd_req,
+				unsigned int which, u16 opcode,
  				u64 offset, u64 length,
  				u64 truncate_size, u32 truncate_seq)
  {
+	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
  	size_t payload_len = 0;
 
  	BUG_ON(opcode != CEPH_OSD_OP_READ && opcode != CEPH_OSD_OP_WRITE);
 
-	osd_req_op_init(op, opcode);
-
  	op->extent.offset = offset;
  	op->extent.length = length;
  	op->extent.truncate_size = truncate_size;
@@ -359,9 +366,15 @@ void osd_req_op_extent_init(struct ceph_osd_req_op
 *op, u16 opcode,
  }
  EXPORT_SYMBOL(osd_req_op_extent_init);
 
-void osd_req_op_extent_update(struct ceph_osd_req_op *op, u64 length)
+void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
+				unsigned int which, u64 length)
  {
-	u64 previous = op->extent.length;
+	struct ceph_osd_req_op *op;
+	u64 previous;
+
+	BUG_ON(which >= osd_req->r_num_ops);
+	op = &osd_req->r_ops[which];
+	previous = op->extent.length;
 
  	if (length == previous)

return; /* Nothing to do */

net/ceph/osd_client.c +7 -6
@@ -372,24 +385,25 @@ void osd_req_op_extent_update(struct
 ceph_osd_req_op *op, u64 length)
  }
  EXPORT_SYMBOL(osd_req_op_extent_update);
 
-void osd_req_op_extent_osd_data(struct ceph_osd_req_op *op,
+void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+				unsigned int which,
  				struct ceph_osd_data *osd_data)
  {
-	op->extent.osd_data = osd_data;
+	BUG_ON(which >= osd_req->r_num_ops);
+	osd_req->r_ops[which].extent.osd_data = osd_data;
  }
  EXPORT_SYMBOL(osd_req_op_extent_osd_data);
 
-void osd_req_op_cls_init(struct ceph_osd_req_op *op, u16 opcode,
-			const char *class, const char *method,
+void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int
 which,
+			u16 opcode, const char *class, const char *method,
  			const void *request_data, size_t request_data_size)
  {
+	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
  	size_t payload_len = 0;
  	size_t size;
 
  	BUG_ON(opcode != CEPH_OSD_OP_CALL);
 
-	osd_req_op_init(op, opcode);
-
  	op->cls.class_name = class;
  	size = strlen(class);
  	BUG_ON(size > (size_t) U8_MAX);
net/ceph/osd_client.c +9 -8
@@ -412,26 +426,28 @@ void osd_req_op_cls_init(struct ceph_osd_req_op
 *op, u16 opcode,
  	op->payload_len = payload_len;
  }
  EXPORT_SYMBOL(osd_req_op_cls_init);
-
-void osd_req_op_cls_response_data(struct ceph_osd_req_op *op,
+void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+				unsigned int which,
  				struct ceph_osd_data *response_data)
  {
-	op->cls.response_data = response_data;
+	BUG_ON(which >= osd_req->r_num_ops);
+	osd_req->r_ops[which].cls.response_data = response_data;
  }
  EXPORT_SYMBOL(osd_req_op_cls_response_data);
 
-void osd_req_op_watch_init(struct ceph_osd_req_op *op, u16 opcode,
+void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
+				unsigned int which, u16 opcode,
  				u64 cookie, u64 version, int flag)
  {
-	BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);
+	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
 
-	osd_req_op_init(op, opcode);
+	BUG_ON(opcode != CEPH_OSD_OP_NOTIFY_ACK && opcode != CEPH_OSD_OP_WATCH);
 
  	op->watch.cookie = cookie;
  	/* op->watch.ver = version; */	/* XXX 3847 */
  	op->watch.ver = cpu_to_le64(version);
  	if (opcode == CEPH_OSD_OP_WATCH && flag)
-		op->watch.flag = (u8) 1;
+		op->watch.flag = (u8)1;
  }
  EXPORT_SYMBOL(osd_req_op_watch_init);
@@ -629,7 +645,6 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  {
  	struct ceph_osd_request *req;
  	struct ceph_osd_data *osd_data;
-	struct ceph_osd_req_op *op;
  	u64 objnum = 0;
  	u64 objoff = 0;
  	u64 objlen = 0;
net/ceph/osd_client.c +2 -3
@@ -665,10 +680,9 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  			truncate_size = object_size;
  	}
 
-	op = &req->r_ops[0];
-	osd_req_op_extent_init(op, opcode, objoff, objlen,
+	osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
  				truncate_size, truncate_seq);
-	osd_req_op_extent_osd_data(op, osd_data);
+	osd_req_op_extent_osd_data(req, 0, osd_data);
 
  	/*

* A second op in the ops array means the caller wants to

net/ceph/osd_client.c +1 -1
@@ -676,7 +690,7 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  	 * osd will flush data quickly.
  	 */
  	if (num_ops > 1)
-		osd_req_op_init(++op, CEPH_OSD_OP_STARTSYNC);
+		osd_req_op_init(req, 1, CEPH_OSD_OP_STARTSYNC);
 
  	req->r_file_layout = *layout;  /* keep a copy */
author
#23 lore raw

PATCH 12/20 rbd: don't set data in rbd_osd_req_format_op()

In reply to Alex Elder

Currently an object request has its osd request's data field set in rbd_osd_req_format_op(). That assumes a single osd op per object request, and that won't be the case for long.

Move the code that sets this out and into the caller.

Rename rbd_osd_req_format_op() to be just rbd_osd_req_format(), removing the notion that it's doing anything op-specific.

This and the next patch resolve:

    http://tracker.ceph.com/issues/4658
---
 drivers/block/rbd.c |   55
+++++++++++++++++++++++----------------------------
 1 file changed, 25 insertions(+), 30 deletions(-)
drivers/block/rbd.c +0 -0
index 7a62327..3b90283 100644
@@ -1311,12 +1311,11 @@ static void rbd_osd_req_callback(struct

ceph_osd_request *osd_req,
rbd_obj_request_complete(obj_request);
}

-static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request, +static void rbd_osd_req_format(struct rbd_obj_request *obj_request,
bool write_request)
{
struct rbd_img_request *img_request = obj_request->img_request; struct ceph_osd_request *osd_req = obj_request->osd_req;
- struct ceph_osd_data *osd_data = NULL;
struct ceph_snap_context *snapc = NULL;
u64 snap_id = CEPH_NOSNAP;
struct timespec *mtime = NULL;
@@ -1325,28 +1324,12 @@ static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
rbd_assert(osd_req != NULL);

if (write_request) {
- osd_data = &osd_req->r_data_out;
now = CURRENT_TIME;
mtime = &now;
if (img_request)
snapc = img_request->snapc;
- } else {
- osd_data = &osd_req->r_data_in;
- if (img_request)
- snap_id = img_request->snap_id;
- }
- if (obj_request->type != OBJ_REQUEST_NODATA) {
- /*
- * If it has data, it's either a object class method
- * call (cls) or it's an extent operation.
- */
- /* XXX This use of the ops array goes away in the next patch */
- if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
- osd_req_op_cls_response_data(obj_request->osd_req, 0,
- osd_data);
- else
- osd_req_op_extent_osd_data(obj_request->osd_req, 0,
- osd_data);
+ } else if (img_request) {
+ snap_id = img_request->snap_id;
}
ceph_osdc_build_request(osd_req, obj_request->offset,
snapc, snap_id, mtime);
@@ -1576,6 +1559,8 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
resid = img_request->length;
rbd_assert(resid > 0);
while (resid) {
+ struct ceph_osd_request *osd_req;
+ struct ceph_osd_data *osd_data;
const char *object_name;
unsigned int clone_size;
u64 offset;
@@ -1601,14 +1586,18 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
if (!obj_request->bio_list)
goto out_partial;

- obj_request->osd_req = rbd_osd_req_create(rbd_dev,
- write_request, obj_request);
- if (!obj_request->osd_req)
+ osd_req = rbd_osd_req_create(rbd_dev, write_request,
+ obj_request);
+ if (!osd_req)
goto out_partial;
+ obj_request->osd_req = osd_req;

- osd_req_op_extent_init(obj_request->osd_req, 0,
- opcode, offset, length, 0, 0);
- rbd_osd_req_format_op(obj_request, write_request);
+ osd_data = write_request ? &osd_req->r_data_out
+ : &osd_req->r_data_in;
+ osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
+ 0, 0);
+ osd_req_op_extent_osd_data(osd_req, 0, osd_data);
+ rbd_osd_req_format(obj_request, write_request);

/* status and version are initially zero-filled */

@@ -1724,7 +1713,7 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,

osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
notify_id, ver, 0);
- rbd_osd_req_format_op(obj_request, false);
+ rbd_osd_req_format(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc; obj_request->callback = rbd_obj_request_put;
@@ -1790,7 +1779,7 @@ static int rbd_dev_header_watch_sync(struct rbd_device *rbd_dev, int start)
osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
rbd_dev->watch_event->cookie, rbd_dev->header.obj_version, start);
- rbd_osd_req_format_op(obj_request, true);
+ rbd_osd_req_format(obj_request, true);

if (start)
ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
@@ -1849,6 +1838,7 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
u64 *version)
{
struct rbd_obj_request *obj_request;
+ struct ceph_osd_data *osd_data;
struct ceph_osd_client *osdc;
struct page **pages;
u32 page_count;
@@ -1879,10 +1869,12 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

+ osd_data = &obj_request->osd_req->r_data_in;
osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
class_name, method_name, outbound, outbound_size);
- rbd_osd_req_format_op(obj_request, false);
+ osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
+ rbd_osd_req_format(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc;
ret = rbd_obj_request_submit(osdc, obj_request);
@@ -2061,6 +2053,7 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,

{
struct rbd_obj_request *obj_request;
+ struct ceph_osd_data *osd_data;
struct ceph_osd_client *osdc;
struct page **pages = NULL;
u32 page_count;
@@ -2085,9 +2078,11 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

+ osd_data = &obj_request->osd_req->r_data_in;
osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
offset, length, 0, 0);
- rbd_osd_req_format_op(obj_request, false);
+ osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
+ rbd_osd_req_format(obj_request, false);

osdc = &rbd_dev->rbd_client->client->osdc; ret = rbd_obj_request_submit(osdc, obj_request);

Signature · 1 line
1.7.9.5
Reviewed-by on v1 12/20
#24 lore raw

Re: [PATCH 12/20] rbd: don't set data in rbd_osd_req_format_op()

Quoted 173 lines · Alex Elder

Currently an object request has its osd request's data field set in rbd_osd_req_format_op(). That assumes a single osd op per object request, and that won't be the case for long.

Move the code that sets this out and into the caller.

Rename rbd_osd_req_format_op() to be just rbd_osd_req_format(), removing the notion that it's doing anything op-specific.

This and the next patch resolve:

     http://tracker.ceph.com/issues/4658
---
  drivers/block/rbd.c |   55
+++++++++++++++++++++++----------------------------
  1 file changed, 25 insertions(+), 30 deletions(-)
drivers/block/rbd.c +1 -2
index 7a62327..3b90283 100644
@@ -1311,12 +1311,11 @@ static void rbd_osd_req_callback(struct
 ceph_osd_request *osd_req,
  		rbd_obj_request_complete(obj_request);
  }
 
-static void rbd_osd_req_format_op(struct rbd_obj_request *obj_request,
+static void rbd_osd_req_format(struct rbd_obj_request *obj_request,
  					bool write_request)
  {
  	struct rbd_img_request *img_request = obj_request->img_request;
  	struct ceph_osd_request *osd_req = obj_request->osd_req;
-	struct ceph_osd_data *osd_data = NULL;
  	struct ceph_snap_context *snapc = NULL;
  	u64 snap_id = CEPH_NOSNAP;
  	struct timespec *mtime = NULL;
drivers/block/rbd.c +2 -18
@@ -1325,28 +1324,12 @@ static void rbd_osd_req_format_op(struct
 rbd_obj_request *obj_request,
  	rbd_assert(osd_req != NULL);
 
  	if (write_request) {
-		osd_data = &osd_req->r_data_out;
  		now = CURRENT_TIME;
  		mtime = &now;
  		if (img_request)
  			snapc = img_request->snapc;
-	} else {
-		osd_data = &osd_req->r_data_in;
-		if (img_request)
-			snap_id = img_request->snap_id;
-	}
-	if (obj_request->type != OBJ_REQUEST_NODATA) {
-		/*
-		 * If it has data, it's either a object class method
-		 * call (cls) or it's an extent operation.
-		 */
-		/* XXX This use of the ops array goes away in the next patch */
-		if (obj_request->osd_req->r_ops[0].op == CEPH_OSD_OP_CALL)
-			osd_req_op_cls_response_data(obj_request->osd_req, 0,
-						osd_data);
-		else
-			osd_req_op_extent_osd_data(obj_request->osd_req, 0,
-						osd_data);
+	} else if (img_request) {
+		snap_id = img_request->snap_id;
  	}
  	ceph_osdc_build_request(osd_req, obj_request->offset,

snapc, snap_id, mtime);

drivers/block/rbd.c +2 -0
@@ -1576,6 +1559,8 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  	resid = img_request->length;
  	rbd_assert(resid > 0);
  	while (resid) {
+		struct ceph_osd_request *osd_req;
+		struct ceph_osd_data *osd_data;
  		const char *object_name;
  		unsigned int clone_size;

u64 offset;

drivers/block/rbd.c +11 -7
@@ -1601,14 +1586,18 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  		if (!obj_request->bio_list)
  			goto out_partial;
 
-		obj_request->osd_req = rbd_osd_req_create(rbd_dev,
-						write_request, obj_request);
-		if (!obj_request->osd_req)
+		osd_req = rbd_osd_req_create(rbd_dev, write_request,
+						obj_request);
+		if (!osd_req)
  			goto out_partial;
+		obj_request->osd_req = osd_req;
 
-		osd_req_op_extent_init(obj_request->osd_req, 0,
-					opcode, offset, length, 0, 0);
-		rbd_osd_req_format_op(obj_request, write_request);
+		osd_data = write_request ? &osd_req->r_data_out
+					 : &osd_req->r_data_in;
+		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
+						0, 0);
+		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
+		rbd_osd_req_format(obj_request, write_request);
 
  		/* status and version are initially zero-filled */
@@ -1724,7 +1713,7 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
 
  	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
  					notify_id, ver, 0);
-	rbd_osd_req_format_op(obj_request, false);
+	rbd_osd_req_format(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	obj_request->callback = rbd_obj_request_put;
drivers/block/rbd.c +1 -1
@@ -1790,7 +1779,7 @@ static int rbd_dev_header_watch_sync(struct
 rbd_device *rbd_dev, int start)
  	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
  				rbd_dev->watch_event->cookie,
  				rbd_dev->header.obj_version, start);
-	rbd_osd_req_format_op(obj_request, true);
+	rbd_osd_req_format(obj_request, true);
 
  	if (start)
  		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
drivers/block/rbd.c +1 -0
@@ -1849,6 +1838,7 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  			     u64 *version)
  {
  	struct rbd_obj_request *obj_request;
+	struct ceph_osd_data *osd_data;
  	struct ceph_osd_client *osdc;
  	struct page **pages;

u32 page_count;

drivers/block/rbd.c +3 -1
@@ -1879,10 +1869,12 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
+	osd_data = &obj_request->osd_req->r_data_in;
  	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
  					class_name, method_name,
  					outbound, outbound_size);
-	rbd_osd_req_format_op(obj_request, false);
+	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
+	rbd_osd_req_format(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	ret = rbd_obj_request_submit(osdc, obj_request);
drivers/block/rbd.c +1 -0
@@ -2061,6 +2053,7 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
 
  {
  	struct rbd_obj_request *obj_request;
+	struct ceph_osd_data *osd_data;
  	struct ceph_osd_client *osdc;
  	struct page **pages = NULL;

u32 page_count;

drivers/block/rbd.c +3 -1
@@ -2085,9 +2078,11 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
+	osd_data = &obj_request->osd_req->r_data_in;
  	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
  					offset, length, 0, 0);
-	rbd_osd_req_format_op(obj_request, false);
+	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
+	rbd_osd_req_format(obj_request, false);
 
  	osdc = &rbd_dev->rbd_client->client->osdc;
  	ret = rbd_obj_request_submit(osdc, obj_request);
author
#25 lore raw

PATCH 13/20 rbd: separate initialization of osd data

In reply to Alex Elder

The osd data for a request is currently initialized inside rbd_osd_req_create(), but that assumes an object request's data belongs in the osd request's data in or data out field.

There are only three places where requests with data are set up, and it turns out it's easier to call just the osd data init routines directly there rather than handling it in rbd_osd_req_create().

(The real motivation here is moving toward getting rid of the osd request in and out data fields.)

---
 drivers/block/rbd.c |   27 ++++++++-------------------
 1 file changed, 8 insertions(+), 19 deletions(-)
drivers/block/rbd.c +0 -19
index 3b90283..1cd776b 100644
@@ -1344,8 +1344,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
 	struct ceph_snap_context *snapc = NULL;
 	struct ceph_osd_client *osdc;
 	struct ceph_osd_request *osd_req;
-	struct ceph_osd_data *osd_data;
-	u64 offset = obj_request->offset;
 
 	if (img_request) {
 		rbd_assert(img_request->write_request == write_request);
@@ -1359,23 +1357,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
 	osd_req = ceph_osdc_alloc_request(osdc, snapc, 1, false, GFP_ATOMIC);
 	if (!osd_req)
 		return NULL;	/* ENOMEM */
-	osd_data = write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
-
-	rbd_assert(obj_request_type_valid(obj_request->type));
-	switch (obj_request->type) {
-	case OBJ_REQUEST_NODATA:
-		break;		/* Nothing to do */
-	case OBJ_REQUEST_BIO:
-		rbd_assert(obj_request->bio_list != NULL);
-		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
-					obj_request->length);
-		break;
-	case OBJ_REQUEST_PAGES:
-		ceph_osd_data_pages_init(osd_data, obj_request->pages,
-				obj_request->length, offset & ~PAGE_MASK,
-				false, false);
-		break;
-	}
 
 	if (write_request)
 		osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
@@ -1596,6 +1577,8 @@ static int rbd_img_request_fill_bio(struct

rbd_img_request *img_request,
: &osd_req->r_data_in;
osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
0, 0);
+ ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
+ obj_request->length);
osd_req_op_extent_osd_data(osd_req, 0, osd_data); rbd_osd_req_format(obj_request, write_request);

@@ -1873,6 +1856,8 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
class_name, method_name, outbound, outbound_size);
+ ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
+ false, false);
osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data); rbd_osd_req_format(obj_request, false);

@@ -2081,6 +2066,10 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
osd_data = &obj_request->osd_req->r_data_in; osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
offset, length, 0, 0);
+ ceph_osd_data_pages_init(osd_data, obj_request->pages,
+ obj_request->length,
+ obj_request->offset & ~PAGE_MASK,
+ false, false);
osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data); rbd_osd_req_format(obj_request, false);

Signature · 1 line
1.7.9.5
Reviewed-by on v1 13/20
#26 lore raw

Re: [PATCH 13/20] rbd: separate initialization of osd data

Quoted 85 lines · Alex Elder

The osd data for a request is currently initialized inside rbd_osd_req_create(), but that assumes an object request's data belongs in the osd request's data in or data out field.

There are only three places where requests with data are set up, and it turns out it's easier to call just the osd data init routines directly there rather than handling it in rbd_osd_req_create().

(The real motivation here is moving toward getting rid of the osd request in and out data fields.)

---
  drivers/block/rbd.c |   27 ++++++++-------------------
  1 file changed, 8 insertions(+), 19 deletions(-)
drivers/block/rbd.c +8 -19
index 3b90283..1cd776b 100644
@@ -1344,8 +1344,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
  	struct ceph_snap_context *snapc = NULL;
  	struct ceph_osd_client *osdc;
  	struct ceph_osd_request *osd_req;
-	struct ceph_osd_data *osd_data;
-	u64 offset = obj_request->offset;
 
  	if (img_request) {
  		rbd_assert(img_request->write_request == write_request);
@@ -1359,23 +1357,6 @@ static struct ceph_osd_request *rbd_osd_req_create(
  	osd_req = ceph_osdc_alloc_request(osdc, snapc, 1, false, GFP_ATOMIC);
  	if (!osd_req)
  		return NULL;	/* ENOMEM */
-	osd_data = write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
-
-	rbd_assert(obj_request_type_valid(obj_request->type));
-	switch (obj_request->type) {
-	case OBJ_REQUEST_NODATA:
-		break;		/* Nothing to do */
-	case OBJ_REQUEST_BIO:
-		rbd_assert(obj_request->bio_list != NULL);
-		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
-					obj_request->length);
-		break;
-	case OBJ_REQUEST_PAGES:
-		ceph_osd_data_pages_init(osd_data, obj_request->pages,
-				obj_request->length, offset & ~PAGE_MASK,
-				false, false);
-		break;
-	}
 
  	if (write_request)
  		osd_req->r_flags = CEPH_OSD_FLAG_WRITE | CEPH_OSD_FLAG_ONDISK;
@@ -1596,6 +1577,8 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  					 : &osd_req->r_data_in;
  		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
  						0, 0);
+		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
+					obj_request->length);
  		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
  		rbd_osd_req_format(obj_request, write_request);
@@ -1873,6 +1856,8 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
  					class_name, method_name,
  					outbound, outbound_size);
+	ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
+					false, false);
  	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
  	rbd_osd_req_format(obj_request, false);
@@ -2081,6 +2066,10 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  	osd_data = &obj_request->osd_req->r_data_in;
  	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
  					offset, length, 0, 0);
+	ceph_osd_data_pages_init(osd_data, obj_request->pages,
+					obj_request->length,
+					obj_request->offset & ~PAGE_MASK,
+					false, false);
  	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
  	rbd_osd_req_format(obj_request, false);
author
#27 lore raw

PATCH 14/20 rbd: rearrange some code for consistency

In reply to Alex Elder

This patch just trivially moves around some code for consistency.

In preparation for initializing osd request data fields in ceph_osdc_build_request(), I wanted to verify that rbd did in fact call that immediatley before it called ceph_osdc_start_request(). It was true (although image requests are built in a group and then started as a group). But I made the changes here just to make it more obvious, by making all of the calls follow a common sequence:

	osd_req_op_<optype>_init();
	ceph_osd_data_<type>_init()
	osd_req_op_<optype>_<datafield>()
	rbd_osd_req_format()
	...
	ret = rbd_obj_request_submit()

I moved the initialization of the callback for image object requests into rbd_img_request_fill_bio(), again, for consistency. To avoid a forward reference, I moved the definition of rbd_img_obj_callback() up in the file.

---
 drivers/block/rbd.c |  129
+++++++++++++++++++++++++--------------------------
 1 file changed, 63 insertions(+), 66 deletions(-)
drivers/block/rbd.c +0 -0
index 1cd776b..3e8e6d5 100644
@@ -1519,6 +1519,57 @@ static void rbd_img_request_destroy(struct kref

*kref)
kfree(img_request);
}

+static void rbd_img_obj_callback(struct rbd_obj_request *obj_request) +{
+ struct rbd_img_request *img_request;
+ u32 which = obj_request->which;
+ bool more = true;
+
+ img_request = obj_request->img_request;
+
+ dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
+ rbd_assert(img_request != NULL);
+ rbd_assert(img_request->rq != NULL);
+ rbd_assert(img_request->obj_request_count > 0);
+ rbd_assert(which != BAD_WHICH);
+ rbd_assert(which < img_request->obj_request_count);
+ rbd_assert(which >= img_request->next_completion);
+
+ spin_lock_irq(&img_request->completion_lock);
+ if (which != img_request->next_completion)
+ goto out;
+
+ for_each_obj_request_from(img_request, obj_request) {
+ unsigned int xferred;
+ int result;
+
+ rbd_assert(more);
+ rbd_assert(which < img_request->obj_request_count);
+
+ if (!obj_request_done_test(obj_request))
+ break;
+
+ rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
+ xferred = (unsigned int) obj_request->xferred;
+ result = (int) obj_request->result;
+ if (result)
+ rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
+ img_request->write_request ? "write" : "read",
+ result, xferred);
+
+ more = blk_end_request(img_request->rq, result, xferred);
+ which++;
+ }
+
+ rbd_assert(more ^ (which == img_request->obj_request_count));
+ img_request->next_completion = which;
+out:
+ spin_unlock_irq(&img_request->completion_lock);
+
+ if (!more)
+ rbd_img_request_complete(img_request);
+}
+
static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
struct bio *bio_list)
{
@@ -1572,6 +1623,7 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
if (!osd_req)
goto out_partial;
obj_request->osd_req = osd_req;
+ obj_request->callback = rbd_img_obj_callback;

osd_data = write_request ? &osd_req->r_data_out
: &osd_req->r_data_in;
@@ -1582,8 +1634,6 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
osd_req_op_extent_osd_data(osd_req, 0, osd_data); rbd_osd_req_format(obj_request, write_request);

- /* status and version are initially zero-filled */
-
rbd_img_obj_request_add(img_request, obj_request);

image_offset += length;
@@ -1601,57 +1651,6 @@ out_unwind:
return -ENOMEM;
}

-static void rbd_img_obj_callback(struct rbd_obj_request *obj_request) -{
- struct rbd_img_request *img_request;
- u32 which = obj_request->which;
- bool more = true;
-
- img_request = obj_request->img_request;
-
- dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
- rbd_assert(img_request != NULL);
- rbd_assert(img_request->rq != NULL);
- rbd_assert(img_request->obj_request_count > 0);
- rbd_assert(which != BAD_WHICH);
- rbd_assert(which < img_request->obj_request_count);
- rbd_assert(which >= img_request->next_completion);
-
- spin_lock_irq(&img_request->completion_lock);
- if (which != img_request->next_completion)
- goto out;
-
- for_each_obj_request_from(img_request, obj_request) {
- unsigned int xferred;
- int result;
-
- rbd_assert(more);
- rbd_assert(which < img_request->obj_request_count);
-
- if (!obj_request_done_test(obj_request))
- break;
-
- rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
- xferred = (unsigned int) obj_request->xferred;
- result = (int) obj_request->result;
- if (result)
- rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
- img_request->write_request ? "write" : "read",
- result, xferred);
-
- more = blk_end_request(img_request->rq, result, xferred);
- which++;
- }
-
- rbd_assert(more ^ (which == img_request->obj_request_count));
- img_request->next_completion = which;
-out:
- spin_unlock_irq(&img_request->completion_lock);
-
- if (!more)
- rbd_img_request_complete(img_request);
-}
-
static int rbd_img_request_submit(struct rbd_img_request *img_request) {
struct rbd_device *rbd_dev = img_request->rbd_dev;
@@ -1662,7 +1661,6 @@ static int rbd_img_request_submit(struct rbd_img_request *img_request)
for_each_obj_request(img_request, obj_request) {
int ret;

- obj_request->callback = rbd_img_obj_callback;
ret = rbd_obj_request_submit(osdc, obj_request); if (ret)
return ret;
@@ -1681,8 +1679,9 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,
u64 ver, u64 notify_id)
{
struct rbd_obj_request *obj_request;
- struct ceph_osd_client *osdc;
+ struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
int ret;
+ osdc = &rbd_dev->rbd_client->client->osdc;

obj_request = rbd_obj_request_create(rbd_dev->header_name, 0, 0,
OBJ_REQUEST_NODATA);
@@ -1693,13 +1692,12 @@ static int rbd_obj_notify_ack(struct rbd_device *rbd_dev,
obj_request->osd_req = rbd_osd_req_create(rbd_dev, false, obj_request); if (!obj_request->osd_req)
goto out;
+ obj_request->callback = rbd_obj_request_put;

osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
notify_id, ver, 0);
rbd_osd_req_format(obj_request, false);

- osdc = &rbd_dev->rbd_client->client->osdc;
- obj_request->callback = rbd_obj_request_put;
ret = rbd_obj_request_submit(osdc, obj_request);
out:
if (ret)
@@ -1759,16 +1757,17 @@ static int rbd_dev_header_watch_sync(struct rbd_device *rbd_dev, int start)
if (!obj_request->osd_req)
goto out_cancel;

- osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
- rbd_dev->watch_event->cookie,
- rbd_dev->header.obj_version, start);
- rbd_osd_req_format(obj_request, true);
-
if (start)
ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
else
ceph_osdc_unregister_linger_request(osdc,
rbd_dev->watch_request->osd_req);
+
+ osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
+ rbd_dev->watch_event->cookie,
+ rbd_dev->header.obj_version, start);
+ rbd_osd_req_format(obj_request, true);
+
ret = rbd_obj_request_submit(osdc, obj_request);
if (ret)
goto out_cancel;
@@ -1820,9 +1819,9 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
size_t inbound_size,
u64 *version)
{
+ struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
struct rbd_obj_request *obj_request;
struct ceph_osd_data *osd_data;
- struct ceph_osd_client *osdc;
struct page **pages;
u32 page_count;
int ret;
@@ -1861,7 +1860,6 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data); rbd_osd_req_format(obj_request, false);

- osdc = &rbd_dev->rbd_client->client->osdc;
ret = rbd_obj_request_submit(osdc, obj_request); if (ret)
goto out;
@@ -2037,9 +2035,9 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
char *buf, u64 *version)

{
+ struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
struct rbd_obj_request *obj_request;
struct ceph_osd_data *osd_data;
- struct ceph_osd_client *osdc;
struct page **pages = NULL;
u32 page_count;
size_t size;
@@ -2073,7 +2071,6 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data); rbd_osd_req_format(obj_request, false);

- osdc = &rbd_dev->rbd_client->client->osdc;
ret = rbd_obj_request_submit(osdc, obj_request); if (ret)
goto out;

Signature · 1 line
1.7.9.5
Reviewed-by on v1 14/20
#28 lore raw

Re: [PATCH 14/20] rbd: rearrange some code for consistency

Just a couple nits.

Quoted 5 lines · Alex Elder

This patch just trivially moves around some code for consistency.

In preparation for initializing osd request data fields in ceph_osdc_build_request(), I wanted to verify that rbd did in fact call that immediatley before it called ceph_osdc_start_request().

typo in immediately

Quoted 173 lines

It was true (although image requests are built in a group and then started as a group). But I made the changes here just to make it more obvious, by making all of the calls follow a common sequence:

	osd_req_op_<optype>_init();
	ceph_osd_data_<type>_init()
	osd_req_op_<optype>_<datafield>()
	rbd_osd_req_format()
	...
	ret = rbd_obj_request_submit()

I moved the initialization of the callback for image object requests into rbd_img_request_fill_bio(), again, for consistency. To avoid a forward reference, I moved the definition of rbd_img_obj_callback() up in the file.

---
drivers/block/rbd.c | 129
+++++++++++++++++++++++++--------------------------

1 file changed, 63 insertions(+), 66 deletions(-)
drivers/block/rbd.c +51 -0
index 1cd776b..3e8e6d5 100644
@@ -1519,6 +1519,57 @@ static void rbd_img_request_destroy(struct kref
 *kref)
 	kfree(img_request);
 }
 
+static void rbd_img_obj_callback(struct rbd_obj_request *obj_request)
+{
+	struct rbd_img_request *img_request;
+	u32 which = obj_request->which;
+	bool more = true;
+
+	img_request = obj_request->img_request;
+
+	dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
+	rbd_assert(img_request != NULL);
+	rbd_assert(img_request->rq != NULL);
+	rbd_assert(img_request->obj_request_count > 0);
+	rbd_assert(which != BAD_WHICH);
+	rbd_assert(which < img_request->obj_request_count);
+	rbd_assert(which >= img_request->next_completion);
+
+	spin_lock_irq(&img_request->completion_lock);
+	if (which != img_request->next_completion)
+		goto out;
+
+	for_each_obj_request_from(img_request, obj_request) {
+		unsigned int xferred;
+		int result;
+
+		rbd_assert(more);
+		rbd_assert(which < img_request->obj_request_count);
+
+		if (!obj_request_done_test(obj_request))
+			break;
+
+		rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
+		xferred = (unsigned int) obj_request->xferred;
+		result = (int) obj_request->result;
+		if (result)
+			rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
+				img_request->write_request ? "write" : "read",
+				result, xferred);
+
+		more = blk_end_request(img_request->rq, result, xferred);
+		which++;
+	}
+
+	rbd_assert(more ^ (which == img_request->obj_request_count));
+	img_request->next_completion = which;
+out:
+	spin_unlock_irq(&img_request->completion_lock);
+
+	if (!more)
+		rbd_img_request_complete(img_request);
+}
+

static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
struct bio *bio_list)
{

drivers/block/rbd.c +1 -0
@@ -1572,6 +1623,7 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
 		if (!osd_req)
 			goto out_partial;
 		obj_request->osd_req = osd_req;
+		obj_request->callback = rbd_img_obj_callback;
 
 		osd_data = write_request ? &osd_req->r_data_out
					 : &osd_req->r_data_in;
drivers/block/rbd.c +0 -2
@@ -1582,8 +1634,6 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
 		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
 		rbd_osd_req_format(obj_request, write_request);
 
-		/* status and version are initially zero-filled */
-
 		rbd_img_obj_request_add(img_request, obj_request);
 
		image_offset += length;
drivers/block/rbd.c +0 -52
@@ -1601,57 +1651,6 @@ out_unwind:
 	return -ENOMEM;
 }
 
-static void rbd_img_obj_callback(struct rbd_obj_request *obj_request)
-{
-	struct rbd_img_request *img_request;
-	u32 which = obj_request->which;
-	bool more = true;
-
-	img_request = obj_request->img_request;
-
-	dout("%s: img %p obj %p\n", __func__, img_request, obj_request);
-	rbd_assert(img_request != NULL);
-	rbd_assert(img_request->rq != NULL);
-	rbd_assert(img_request->obj_request_count > 0);
-	rbd_assert(which != BAD_WHICH);
-	rbd_assert(which < img_request->obj_request_count);
-	rbd_assert(which >= img_request->next_completion);
-
-	spin_lock_irq(&img_request->completion_lock);
-	if (which != img_request->next_completion)
-		goto out;
-
-	for_each_obj_request_from(img_request, obj_request) {
-		unsigned int xferred;
-		int result;
-
-		rbd_assert(more);
-		rbd_assert(which < img_request->obj_request_count);
-
-		if (!obj_request_done_test(obj_request))
-			break;
-
-		rbd_assert(obj_request->xferred <= (u64) UINT_MAX);
-		xferred = (unsigned int) obj_request->xferred;
-		result = (int) obj_request->result;
-		if (result)
-			rbd_warn(NULL, "obj_request %s result %d xferred %u\n",
-				img_request->write_request ? "write" : "read",
-				result, xferred);
-
-		more = blk_end_request(img_request->rq, result, xferred);
-		which++;
-	}
-
-	rbd_assert(more ^ (which == img_request->obj_request_count));
-	img_request->next_completion = which;
-out:
-	spin_unlock_irq(&img_request->completion_lock);
-
-	if (!more)
-		rbd_img_request_complete(img_request);
-}
-
 static int rbd_img_request_submit(struct rbd_img_request *img_request)
 {
 	struct rbd_device *rbd_dev = img_request->rbd_dev;
@@ -1662,7 +1661,6 @@ static int rbd_img_request_submit(struct
 rbd_img_request *img_request)
 	for_each_obj_request(img_request, obj_request) {
 		int ret;
 
-		obj_request->callback = rbd_img_obj_callback;
 		ret = rbd_obj_request_submit(osdc, obj_request);
 		if (ret)
			return ret;

drivers/block/rbd.c static int rbd_obj_notify_ack(struct rbd_device · line 1685

Quoted 8 lines
drivers/block/rbd.c +2 -1
@@ -1681,8 +1679,9 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
 				   u64 ver, u64 notify_id)
 {
 	struct rbd_obj_request *obj_request;
-	struct ceph_osd_client *osdc;
+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	int ret;
+	osdc = &rbd_dev->rbd_client->client->osdc;

This is the same as two lines up.

Quoted 86 lines
drivers/block/rbd.c +1 -2
 	obj_request = rbd_obj_request_create(rbd_dev->header_name, 0, 0,
 							OBJ_REQUEST_NODATA);
@@ -1693,13 +1692,12 @@ static int rbd_obj_notify_ack(struct rbd_device
 *rbd_dev,
 	obj_request->osd_req = rbd_osd_req_create(rbd_dev, false,
 obj_request);
 	if (!obj_request->osd_req)
 		goto out;
+	obj_request->callback = rbd_obj_request_put;
 
 	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_NOTIFY_ACK,
 					notify_id, ver, 0);
 	rbd_osd_req_format(obj_request, false);
 
-	osdc = &rbd_dev->rbd_client->client->osdc;
-	obj_request->callback = rbd_obj_request_put;
 	ret = rbd_obj_request_submit(osdc, obj_request);

out:
if (ret)

drivers/block/rbd.c +6 -5
@@ -1759,16 +1757,17 @@ static int rbd_dev_header_watch_sync(struct
 rbd_device *rbd_dev, int start)
 	if (!obj_request->osd_req)
 		goto out_cancel;
 
-	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
-				rbd_dev->watch_event->cookie,
-				rbd_dev->header.obj_version, start);
-	rbd_osd_req_format(obj_request, true);
-
 	if (start)
 		ceph_osdc_set_request_linger(osdc, obj_request->osd_req);
 	else
 		ceph_osdc_unregister_linger_request(osdc,
 					rbd_dev->watch_request->osd_req);
+
+	osd_req_op_watch_init(obj_request->osd_req, 0, CEPH_OSD_OP_WATCH,
+				rbd_dev->watch_event->cookie,
+				rbd_dev->header.obj_version, start);
+	rbd_osd_req_format(obj_request, true);
+
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
		goto out_cancel;
drivers/block/rbd.c +1 -1
@@ -1820,9 +1819,9 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
 			     size_t inbound_size,
 			     u64 *version)
 {
+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
 	struct ceph_osd_data *osd_data;
-	struct ceph_osd_client *osdc;
 	struct page **pages;
 	u32 page_count;
	int ret;
drivers/block/rbd.c +0 -1
@@ -1861,7 +1860,6 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
 	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);
 
-	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
		goto out;
drivers/block/rbd.c +1 -1
@@ -2037,9 +2035,9 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
 				char *buf, u64 *version)
 
 {
+	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
 	struct rbd_obj_request *obj_request;
 	struct ceph_osd_data *osd_data;
-	struct ceph_osd_client *osdc;
 	struct page **pages = NULL;
 	u32 page_count;
	size_t size;
drivers/block/rbd.c +0 -1
@@ -2073,7 +2071,6 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
 	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
 	rbd_osd_req_format(obj_request, false);
 
-	osdc = &rbd_dev->rbd_client->client->osdc;
 	ret = rbd_obj_request_submit(osdc, obj_request);
 	if (ret)
		goto out;
author
#29 lore raw

PATCH 15/20 libceph: format class info at init time

In reply to Alex Elder

An object class method is formatted using a pagelist which contains the class name, the method name, and the data concatenated into an osd request's outbound data.

Currently when a class op is initialized in osd_req_op_cls_init(), the lengths of and pointers to these three items are recorded. Later, when the op is getting formatted into the request message, a new pagelist is created and that is when these items get copied into the pagelist.

This patch makes it so the pagelist to hold these items is created when the op is initialized instead.

---
 include/linux/ceph/osd_client.h |    3 ++-
 net/ceph/osd_client.c           |   29 +++++++++++++++--------------
 2 files changed, 17 insertions(+), 15 deletions(-)
(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index 144d57c..71c4157 100644

include/linux/ceph/osd_client.h +2 -1
@@ -93,8 +93,9 @@ struct ceph_osd_req_op {
 			const char *class_name;
 			const char *method_name;
 			const void *request_data;
-			u32 request_data_len;
+			struct ceph_osd_data *request_info;
 			struct ceph_osd_data *response_data;
+			u32 request_data_len;
 			__u8 class_len;
 			__u8 method_len;
 			__u8 argc;
net/ceph/osd_client.c +0 -0
index e698ccf..6c6d7b8 100644
@@ -399,28 +399,39 @@ void osd_req_op_cls_init(struct ceph_osd_request

*osd_req, unsigned int which,
const void *request_data, size_t request_data_size)
{
struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
+ struct ceph_pagelist *pagelist;
size_t payload_len = 0;
size_t size;

BUG_ON(opcode != CEPH_OSD_OP_CALL);

+ pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
+ BUG_ON(!pagelist);
+ ceph_pagelist_init(pagelist);
+
op->cls.class_name = class;
size = strlen(class);
BUG_ON(size > (size_t) U8_MAX);
op->cls.class_len = size;
+ ceph_pagelist_append(pagelist, class, size);
payload_len += size;

op->cls.method_name = method;
size = strlen(method);
BUG_ON(size > (size_t) U8_MAX);
op->cls.method_len = size;
+ ceph_pagelist_append(pagelist, method, size);
payload_len += size;

op->cls.request_data = request_data; BUG_ON(request_data_size > (size_t) U32_MAX); op->cls.request_data_len = (u32) request_data_size;
+ ceph_pagelist_append(pagelist, request_data, request_data_size);
payload_len += request_data_size;

+ op->cls.request_info = &osd_req->r_data_out;
+ ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
+
op->cls.argc = 0; /* currently unused */

op->payload_len = payload_len;
@@ -456,7 +467,6 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
{
struct ceph_osd_req_op *src;
u64 request_data_len = 0;
- struct ceph_pagelist *pagelist;

BUG_ON(which >= req->r_num_ops);
src = &req->r_ops[which];
@@ -485,23 +495,14 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
WARN_ON(src->extent.osd_data != &req->r_data_in);
break;
case CEPH_OSD_OP_CALL:
- pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
- BUG_ON(!pagelist);
- ceph_pagelist_init(pagelist);
-
dst->cls.class_len = src->cls.class_len; dst->cls.method_len = src->cls.method_len; dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
- ceph_pagelist_append(pagelist, src->cls.class_name,
- src->cls.class_len);
- ceph_pagelist_append(pagelist, src->cls.method_name,
- src->cls.method_len);
- ceph_pagelist_append(pagelist, src->cls.request_data,
- src->cls.request_data_len);
- ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
-
WARN_ON(src->cls.response_data != &req->r_data_in);
- request_data_len = pagelist->length;
+ WARN_ON(src->cls.request_info != &req->r_data_out);
+ BUG_ON(src->cls.request_info->type !=
+ CEPH_OSD_DATA_TYPE_PAGELIST);
+ request_data_len = src->cls.request_info->pagelist->length;
break;
case CEPH_OSD_OP_STARTSYNC:
break;

Signature · 1 line
1.7.9.5
Reviewed-by on v1 15/20
#30 lore raw

Re: [PATCH 15/20] libceph: format class info at init time

Quoted 119 lines · Alex Elder

An object class method is formatted using a pagelist which contains the class name, the method name, and the data concatenated into an osd request's outbound data.

Currently when a class op is initialized in osd_req_op_cls_init(), the lengths of and pointers to these three items are recorded. Later, when the op is getting formatted into the request message, a new pagelist is created and that is when these items get copied into the pagelist.

This patch makes it so the pagelist to hold these items is created when the op is initialized instead.

---
  include/linux/ceph/osd_client.h |    3 ++-
  net/ceph/osd_client.c           |   29 +++++++++++++++--------------
  2 files changed, 17 insertions(+), 15 deletions(-)
include/linux/ceph/osd_client.h +2 -1
index 144d57c..71c4157 100644
@@ -93,8 +93,9 @@ struct ceph_osd_req_op {
  			const char *class_name;
  			const char *method_name;
  			const void *request_data;
-			u32 request_data_len;
+			struct ceph_osd_data *request_info;
  			struct ceph_osd_data *response_data;
+			u32 request_data_len;
  			__u8 class_len;
  			__u8 method_len;
  			__u8 argc;
net/ceph/osd_client.c +11 -0
index e698ccf..6c6d7b8 100644
@@ -399,28 +399,39 @@ void osd_req_op_cls_init(struct ceph_osd_request
 *osd_req, unsigned int which,
  			const void *request_data, size_t request_data_size)
  {
  	struct ceph_osd_req_op *op = osd_req_op_init(osd_req, which, opcode);
+	struct ceph_pagelist *pagelist;
  	size_t payload_len = 0;
  	size_t size;
 
  	BUG_ON(opcode != CEPH_OSD_OP_CALL);
 
+	pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
+	BUG_ON(!pagelist);
+	ceph_pagelist_init(pagelist);
+
  	op->cls.class_name = class;
  	size = strlen(class);
  	BUG_ON(size > (size_t) U8_MAX);
  	op->cls.class_len = size;
+	ceph_pagelist_append(pagelist, class, size);
  	payload_len += size;
 
  	op->cls.method_name = method;
  	size = strlen(method);
  	BUG_ON(size > (size_t) U8_MAX);
  	op->cls.method_len = size;
+	ceph_pagelist_append(pagelist, method, size);
  	payload_len += size;
 
  	op->cls.request_data = request_data;
  	BUG_ON(request_data_size > (size_t) U32_MAX);
  	op->cls.request_data_len = (u32) request_data_size;
+	ceph_pagelist_append(pagelist, request_data, request_data_size);
  	payload_len += request_data_size;
 
+	op->cls.request_info = &osd_req->r_data_out;
+	ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
+
  	op->cls.argc = 0;	/* currently unused */
 
  	op->payload_len = payload_len;
net/ceph/osd_client.c +0 -1
@@ -456,7 +467,6 @@ static u64 osd_req_encode_op(struct ceph_osd_request
 *req,
  {
  	struct ceph_osd_req_op *src;
  	u64 request_data_len = 0;
-	struct ceph_pagelist *pagelist;
 
  	BUG_ON(which >= req->r_num_ops);
  	src = &req->r_ops[which];
net/ceph/osd_client.c +4 -13
@@ -485,23 +495,14 @@ static u64 osd_req_encode_op(struct
 ceph_osd_request *req,
  			WARN_ON(src->extent.osd_data != &req->r_data_in);
  		break;
  	case CEPH_OSD_OP_CALL:
-		pagelist = kmalloc(sizeof (*pagelist), GFP_NOFS);
-		BUG_ON(!pagelist);
-		ceph_pagelist_init(pagelist);
-
  		dst->cls.class_len = src->cls.class_len;
  		dst->cls.method_len = src->cls.method_len;
  		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
-		ceph_pagelist_append(pagelist, src->cls.class_name,
-				     src->cls.class_len);
-		ceph_pagelist_append(pagelist, src->cls.method_name,
-				     src->cls.method_len);
-		ceph_pagelist_append(pagelist, src->cls.request_data,
-				     src->cls.request_data_len);
-		ceph_osd_data_pagelist_init(&req->r_data_out, pagelist);
-
  		WARN_ON(src->cls.response_data != &req->r_data_in);
-		request_data_len = pagelist->length;
+		WARN_ON(src->cls.request_info != &req->r_data_out);
+		BUG_ON(src->cls.request_info->type !=
+					CEPH_OSD_DATA_TYPE_PAGELIST);
+		request_data_len = src->cls.request_info->pagelist->length;
  		break;
  	case CEPH_OSD_OP_STARTSYNC:

break;

author
#31 lore raw

PATCH 16/20 libceph: move ceph_osdc_build_request()

In reply to Alex Elder

This simply moves ceph_osdc_build_request() later in its source file without any change. Done as a separate patch to facilitate review of the change in the next patch.

---
 net/ceph/osd_client.c |  196
++++++++++++++++++++++++-------------------------
 1 file changed, 98 insertions(+), 98 deletions(-)
net/ceph/osd_client.c +0 -0
index 6c6d7b8..f80984e 100644
@@ -526,104 +526,6 @@ static u64 osd_req_encode_op(struct

ceph_osd_request *req,
}

/*
- * build new request AND message
- *
- */
-void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
- struct ceph_snap_context *snapc, u64 snap_id,
- struct timespec *mtime)
-{
- struct ceph_msg *msg = req->r_request;
- void *p;
- size_t msg_size;
- int flags = req->r_flags;
- u64 data_len;
- unsigned int i;
-
- req->r_snapid = snap_id;
- req->r_snapc = ceph_get_snap_context(snapc);
-
- /* encode request */
- msg->hdr.version = cpu_to_le16(4);
-
- p = msg->front.iov_base;
- ceph_encode_32(&p, 1); /* client_inc is always 1 */
- req->r_request_osdmap_epoch = p;
- p += 4;
- req->r_request_flags = p;
- p += 4;
- if (req->r_flags & CEPH_OSD_FLAG_WRITE)
- ceph_encode_timespec(p, mtime);
- p += sizeof(struct ceph_timespec);
- req->r_request_reassert_version = p;
- p += sizeof(struct ceph_eversion); /* will get filled in */
-
- /* oloc */
- ceph_encode_8(&p, 4);
- ceph_encode_8(&p, 4);
- ceph_encode_32(&p, 8 + 4 + 4);
- req->r_request_pool = p;
- p += 8;
- ceph_encode_32(&p, -1); /* preferred */
- ceph_encode_32(&p, 0); /* key len */
-
- ceph_encode_8(&p, 1);
- req->r_request_pgid = p;
- p += 8 + 4;
- ceph_encode_32(&p, -1); /* preferred */
-
- /* oid */
- ceph_encode_32(&p, req->r_oid_len);
- memcpy(p, req->r_oid, req->r_oid_len);
- dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
- p += req->r_oid_len;
-
- /* ops--can imply data */
- ceph_encode_16(&p, (u16)req->r_num_ops);
- data_len = 0;
- for (i = 0; i < req->r_num_ops; i++) {
- data_len += osd_req_encode_op(req, p, i);
- p += sizeof(struct ceph_osd_op);
- }
-
- /* snaps */
- ceph_encode_64(&p, req->r_snapid);
- ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
- ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
- if (req->r_snapc) {
- for (i = 0; i < snapc->num_snaps; i++) {
- ceph_encode_64(&p, req->r_snapc->snaps[i]);
- }
- }
-
- req->r_request_attempts = p;
- p += 4;
-
- /* data */
- if (flags & CEPH_OSD_FLAG_WRITE) {
- u16 data_off;
-
- /*
- * The header "data_off" is a hint to the receiver
- * allowing it to align received data into its
- * buffers such that there's no need to re-copy
- * it before writing it to disk (direct I/O).
- */
- data_off = (u16) (off & 0xffff);
- req->r_request->hdr.data_off = cpu_to_le16(data_off);
- }
- req->r_request->hdr.data_len = cpu_to_le32(data_len);
-
- BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
- msg_size = p - msg->front.iov_base;
- msg->front.iov_len = msg_size;
- msg->hdr.front_len = cpu_to_le32(msg_size);
-
- dout("build_request msg_size was %d\n", (int)msg_size);
-}
-EXPORT_SYMBOL(ceph_osdc_build_request);
-
-/*
* build new request AND message, calculate layout, and adjust file
* extent as needed.
*
@@ -1968,6 +1870,104 @@ static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
}

/*
+ * build new request AND message
+ *
+ */
+void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
+ struct ceph_snap_context *snapc, u64 snap_id,
+ struct timespec *mtime)
+{
+ struct ceph_msg *msg = req->r_request;
+ void *p;
+ size_t msg_size;
+ int flags = req->r_flags;
+ u64 data_len;
+ unsigned int i;
+
+ req->r_snapid = snap_id;
+ req->r_snapc = ceph_get_snap_context(snapc);
+
+ /* encode request */
+ msg->hdr.version = cpu_to_le16(4);
+
+ p = msg->front.iov_base;
+ ceph_encode_32(&p, 1); /* client_inc is always 1 */
+ req->r_request_osdmap_epoch = p;
+ p += 4;
+ req->r_request_flags = p;
+ p += 4;
+ if (req->r_flags & CEPH_OSD_FLAG_WRITE)
+ ceph_encode_timespec(p, mtime);
+ p += sizeof(struct ceph_timespec);
+ req->r_request_reassert_version = p;
+ p += sizeof(struct ceph_eversion); /* will get filled in */
+
+ /* oloc */
+ ceph_encode_8(&p, 4);
+ ceph_encode_8(&p, 4);
+ ceph_encode_32(&p, 8 + 4 + 4);
+ req->r_request_pool = p;
+ p += 8;
+ ceph_encode_32(&p, -1); /* preferred */
+ ceph_encode_32(&p, 0); /* key len */
+
+ ceph_encode_8(&p, 1);
+ req->r_request_pgid = p;
+ p += 8 + 4;
+ ceph_encode_32(&p, -1); /* preferred */
+
+ /* oid */
+ ceph_encode_32(&p, req->r_oid_len);
+ memcpy(p, req->r_oid, req->r_oid_len);
+ dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
+ p += req->r_oid_len;
+
+ /* ops--can imply data */
+ ceph_encode_16(&p, (u16)req->r_num_ops);
+ data_len = 0;
+ for (i = 0; i < req->r_num_ops; i++) {
+ data_len += osd_req_encode_op(req, p, i);
+ p += sizeof(struct ceph_osd_op);
+ }
+
+ /* snaps */
+ ceph_encode_64(&p, req->r_snapid);
+ ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
+ ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
+ if (req->r_snapc) {
+ for (i = 0; i < snapc->num_snaps; i++) {
+ ceph_encode_64(&p, req->r_snapc->snaps[i]);
+ }
+ }
+
+ req->r_request_attempts = p;
+ p += 4;
+
+ /* data */
+ if (flags & CEPH_OSD_FLAG_WRITE) {
+ u16 data_off;
+
+ /*
+ * The header "data_off" is a hint to the receiver
+ * allowing it to align received data into its
+ * buffers such that there's no need to re-copy
+ * it before writing it to disk (direct I/O).
+ */
+ data_off = (u16) (off & 0xffff);
+ req->r_request->hdr.data_off = cpu_to_le16(data_off);
+ }
+ req->r_request->hdr.data_len = cpu_to_le32(data_len);
+
+ BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
+ msg_size = p - msg->front.iov_base;
+ msg->front.iov_len = msg_size;
+ msg->hdr.front_len = cpu_to_le32(msg_size);
+
+ dout("build_request msg_size was %d\n", (int)msg_size);
+}
+EXPORT_SYMBOL(ceph_osdc_build_request);
+
+/*
* Register request, send initial attempt.
*/
int ceph_osdc_start_request(struct ceph_osd_client *osdc,

Signature · 1 line
1.7.9.5
Reviewed-by on v1 16/20
#32 lore raw

Re: [PATCH 16/20] libceph: move ceph_osdc_build_request()

Quoted 227 lines · Alex Elder

This simply moves ceph_osdc_build_request() later in its source file without any change. Done as a separate patch to facilitate review of the change in the next patch.

---
  net/ceph/osd_client.c |  196
++++++++++++++++++++++++-------------------------
  1 file changed, 98 insertions(+), 98 deletions(-)
net/ceph/osd_client.c +0 -98
index 6c6d7b8..f80984e 100644
@@ -526,104 +526,6 @@ static u64 osd_req_encode_op(struct
 ceph_osd_request *req,
  }
 
  /*
- * build new request AND message
- *
- */
-void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
-				struct ceph_snap_context *snapc, u64 snap_id,
-				struct timespec *mtime)
-{
-	struct ceph_msg *msg = req->r_request;
-	void *p;
-	size_t msg_size;
-	int flags = req->r_flags;
-	u64 data_len;
-	unsigned int i;
-
-	req->r_snapid = snap_id;
-	req->r_snapc = ceph_get_snap_context(snapc);
-
-	/* encode request */
-	msg->hdr.version = cpu_to_le16(4);
-
-	p = msg->front.iov_base;
-	ceph_encode_32(&p, 1);   /* client_inc  is always 1 */
-	req->r_request_osdmap_epoch = p;
-	p += 4;
-	req->r_request_flags = p;
-	p += 4;
-	if (req->r_flags & CEPH_OSD_FLAG_WRITE)
-		ceph_encode_timespec(p, mtime);
-	p += sizeof(struct ceph_timespec);
-	req->r_request_reassert_version = p;
-	p += sizeof(struct ceph_eversion); /* will get filled in */
-
-	/* oloc */
-	ceph_encode_8(&p, 4);
-	ceph_encode_8(&p, 4);
-	ceph_encode_32(&p, 8 + 4 + 4);
-	req->r_request_pool = p;
-	p += 8;
-	ceph_encode_32(&p, -1);  /* preferred */
-	ceph_encode_32(&p, 0);   /* key len */
-
-	ceph_encode_8(&p, 1);
-	req->r_request_pgid = p;
-	p += 8 + 4;
-	ceph_encode_32(&p, -1);  /* preferred */
-
-	/* oid */
-	ceph_encode_32(&p, req->r_oid_len);
-	memcpy(p, req->r_oid, req->r_oid_len);
-	dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
-	p += req->r_oid_len;
-
-	/* ops--can imply data */
-	ceph_encode_16(&p, (u16)req->r_num_ops);
-	data_len = 0;
-	for (i = 0; i < req->r_num_ops; i++) {
-		data_len += osd_req_encode_op(req, p, i);
-		p += sizeof(struct ceph_osd_op);
-	}
-
-	/* snaps */
-	ceph_encode_64(&p, req->r_snapid);
-	ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
-	ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
-	if (req->r_snapc) {
-		for (i = 0; i < snapc->num_snaps; i++) {
-			ceph_encode_64(&p, req->r_snapc->snaps[i]);
-		}
-	}
-
-	req->r_request_attempts = p;
-	p += 4;
-
-	/* data */
-	if (flags & CEPH_OSD_FLAG_WRITE) {
-		u16 data_off;
-
-		/*
-		 * The header "data_off" is a hint to the receiver
-		 * allowing it to align received data into its
-		 * buffers such that there's no need to re-copy
-		 * it before writing it to disk (direct I/O).
-		 */
-		data_off = (u16) (off & 0xffff);
-		req->r_request->hdr.data_off = cpu_to_le16(data_off);
-	}
-	req->r_request->hdr.data_len = cpu_to_le32(data_len);
-
-	BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
-	msg_size = p - msg->front.iov_base;
-	msg->front.iov_len = msg_size;
-	msg->hdr.front_len = cpu_to_le32(msg_size);
-
-	dout("build_request msg_size was %d\n", (int)msg_size);
-}
-EXPORT_SYMBOL(ceph_osdc_build_request);
-
-/*
   * build new request AND message, calculate layout, and adjust file
   * extent as needed.
   *
net/ceph/osd_client.c +98 -0
@@ -1968,6 +1870,104 @@ static void ceph_osdc_msg_data_set(struct
 ceph_msg *msg,
  }
 
  /*
+ * build new request AND message
+ *
+ */
+void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
+				struct ceph_snap_context *snapc, u64 snap_id,
+				struct timespec *mtime)
+{
+	struct ceph_msg *msg = req->r_request;
+	void *p;
+	size_t msg_size;
+	int flags = req->r_flags;
+	u64 data_len;
+	unsigned int i;
+
+	req->r_snapid = snap_id;
+	req->r_snapc = ceph_get_snap_context(snapc);
+
+	/* encode request */
+	msg->hdr.version = cpu_to_le16(4);
+
+	p = msg->front.iov_base;
+	ceph_encode_32(&p, 1);   /* client_inc  is always 1 */
+	req->r_request_osdmap_epoch = p;
+	p += 4;
+	req->r_request_flags = p;
+	p += 4;
+	if (req->r_flags & CEPH_OSD_FLAG_WRITE)
+		ceph_encode_timespec(p, mtime);
+	p += sizeof(struct ceph_timespec);
+	req->r_request_reassert_version = p;
+	p += sizeof(struct ceph_eversion); /* will get filled in */
+
+	/* oloc */
+	ceph_encode_8(&p, 4);
+	ceph_encode_8(&p, 4);
+	ceph_encode_32(&p, 8 + 4 + 4);
+	req->r_request_pool = p;
+	p += 8;
+	ceph_encode_32(&p, -1);  /* preferred */
+	ceph_encode_32(&p, 0);   /* key len */
+
+	ceph_encode_8(&p, 1);
+	req->r_request_pgid = p;
+	p += 8 + 4;
+	ceph_encode_32(&p, -1);  /* preferred */
+
+	/* oid */
+	ceph_encode_32(&p, req->r_oid_len);
+	memcpy(p, req->r_oid, req->r_oid_len);
+	dout("oid '%.*s' len %d\n", req->r_oid_len, req->r_oid, req->r_oid_len);
+	p += req->r_oid_len;
+
+	/* ops--can imply data */
+	ceph_encode_16(&p, (u16)req->r_num_ops);
+	data_len = 0;
+	for (i = 0; i < req->r_num_ops; i++) {
+		data_len += osd_req_encode_op(req, p, i);
+		p += sizeof(struct ceph_osd_op);
+	}
+
+	/* snaps */
+	ceph_encode_64(&p, req->r_snapid);
+	ceph_encode_64(&p, req->r_snapc ? req->r_snapc->seq : 0);
+	ceph_encode_32(&p, req->r_snapc ? req->r_snapc->num_snaps : 0);
+	if (req->r_snapc) {
+		for (i = 0; i < snapc->num_snaps; i++) {
+			ceph_encode_64(&p, req->r_snapc->snaps[i]);
+		}
+	}
+
+	req->r_request_attempts = p;
+	p += 4;
+
+	/* data */
+	if (flags & CEPH_OSD_FLAG_WRITE) {
+		u16 data_off;
+
+		/*
+		 * The header "data_off" is a hint to the receiver
+		 * allowing it to align received data into its
+		 * buffers such that there's no need to re-copy
+		 * it before writing it to disk (direct I/O).
+		 */
+		data_off = (u16) (off & 0xffff);
+		req->r_request->hdr.data_off = cpu_to_le16(data_off);
+	}
+	req->r_request->hdr.data_len = cpu_to_le32(data_len);
+
+	BUG_ON(p > msg->front.iov_base + msg->front.iov_len);
+	msg_size = p - msg->front.iov_base;
+	msg->front.iov_len = msg_size;
+	msg->hdr.front_len = cpu_to_le32(msg_size);
+
+	dout("build_request msg_size was %d\n", (int)msg_size);
+}
+EXPORT_SYMBOL(ceph_osdc_build_request);
+
+/*
   * Register request, send initial attempt.
   */

int ceph_osdc_start_request(struct ceph_osd_client *osdc,

author
#33 lore raw

PATCH 17/20 libceph: set message data when building osd request

In reply to Alex Elder

All calls of ceph_osdc_start_request() are preceded (in the case of rbd, almost) immediately by a call to ceph_osdc_build_request().

Move the build calls at the top of ceph_osdc_start_request() out of there and into the ceph_osdc_build_request(). Nothing prevents moving these calls to the top of ceph_osdc_build_request(), either (and we're going to want them there in the next patch) so put them at the top.

This and the next patch are related to:

    http://tracker.ceph.com/issues/4657
---
 net/ceph/osd_client.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
net/ceph/osd_client.c +0 -0
index f80984e..40466ab 100644
@@ -1884,6 +1884,11 @@ void ceph_osdc_build_request(struct

ceph_osd_request *req, u64 off,
u64 data_len;
unsigned int i;

+ /* Set up response incoming data and request outgoing data fields */
+
+ ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
+ ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
+
req->r_snapid = snap_id; req->r_snapc = ceph_get_snap_context(snapc);

@@ -1976,11 +1981,6 @@ int ceph_osdc_start_request(struct ceph_osd_client *osdc,
{
int rc = 0;

- /* Set up response incoming data and request outgoing data fields */
-
- ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
- ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
-
down_read(&osdc->map_sem);
mutex_lock(&osdc->request_mutex);
__register_request(osdc, req);

Signature · 1 line
1.7.9.5
Reviewed-by on v1 17/20
#34 lore raw

Re: [PATCH 17/20] libceph: set message data when building osd request

Quoted 48 lines · Alex Elder

All calls of ceph_osdc_start_request() are preceded (in the case of rbd, almost) immediately by a call to ceph_osdc_build_request().

Move the build calls at the top of ceph_osdc_start_request() out of there and into the ceph_osdc_build_request(). Nothing prevents moving these calls to the top of ceph_osdc_build_request(), either (and we're going to want them there in the next patch) so put them at the top.

This and the next patch are related to:

     http://tracker.ceph.com/issues/4657
---
  net/ceph/osd_client.c |   10 +++++-----
  1 file changed, 5 insertions(+), 5 deletions(-)
net/ceph/osd_client.c +5 -5
index f80984e..40466ab 100644
@@ -1884,6 +1884,11 @@ void ceph_osdc_build_request(struct
 ceph_osd_request *req, u64 off,
  	u64 data_len;
  	unsigned int i;
 
+	/* Set up response incoming data and request outgoing data fields */
+
+	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
+	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
+
  	req->r_snapid = snap_id;
  	req->r_snapc = ceph_get_snap_context(snapc);
@@ -1976,11 +1981,6 @@ int ceph_osdc_start_request(struct
 ceph_osd_client *osdc,
  {
  	int rc = 0;
 
-	/* Set up response incoming data and request outgoing data fields */
-
-	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
-	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
-
  	down_read(&osdc->map_sem);
  	mutex_lock(&osdc->request_mutex);
  	__register_request(osdc, req);
author
#35 lore raw

PATCH 18/20 libceph: combine initializing and setting osd data

In reply to Alex Elder

This ends up being a rather large patch but what it's doing is somewhat straightforward.

Basically, this is replacing two calls with one. The first of the two calls is initializing a struct ceph_osd_data with data (either a page array, a page list, or a bio list); the second is setting an osd request op so it associates that data with one of the op's parameters. In place of those two will be a single function that initializes the op directly.

That means we sort of fan out a set of the needed functions:
- extent ops with pages data
- extent ops with pagelist data
- extent ops with bio list data
and
- class ops with page data for receiving a response

We also have define another one, but it's only used internally:
- class ops with pagelist data for request parameters

Note that we *still* haven't gotten rid of the osd request's r_data_in and r_data_out fields. All the osd ops refer to them for their data. For now, these data fields are pointers assigned to the appropriate r_data_* field when these new functions are called.

---
 drivers/block/rbd.c             |   20 ++---
 fs/ceph/addr.c                  |   12 ++-
 fs/ceph/file.c                  |    3 +-
 include/linux/ceph/osd_client.h |   43 +++++++----
 net/ceph/osd_client.c           |  157
++++++++++++++++++++++++++++++---------
 5 files changed, 162 insertions(+), 73 deletions(-)
drivers/block/rbd.c +0 -0
index 3e8e6d5..7db244e 100644
@@ -1592,7 +1592,6 @@ static int rbd_img_request_fill_bio(struct

rbd_img_request *img_request,
rbd_assert(resid > 0);
while (resid) {
struct ceph_osd_request *osd_req;
- struct ceph_osd_data *osd_data;
const char *object_name;
unsigned int clone_size;
u64 offset;
@@ -1625,13 +1624,10 @@ static int rbd_img_request_fill_bio(struct rbd_img_request *img_request,
obj_request->osd_req = osd_req; obj_request->callback = rbd_img_obj_callback;

- osd_data = write_request ? &osd_req->r_data_out
- : &osd_req->r_data_in;
osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
0, 0);
- ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
- obj_request->length);
- osd_req_op_extent_osd_data(osd_req, 0, osd_data);
+ osd_req_op_extent_osd_data_bio(osd_req, 0, write_request,
+ obj_request->bio_list, obj_request->length);
rbd_osd_req_format(obj_request, write_request);

rbd_img_obj_request_add(img_request, obj_request);
@@ -1821,7 +1817,6 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
{
struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc; struct rbd_obj_request *obj_request;
- struct ceph_osd_data *osd_data;
struct page **pages;
u32 page_count;
int ret;
@@ -1851,13 +1846,12 @@ static int rbd_obj_method_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- osd_data = &obj_request->osd_req->r_data_in;
osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
class_name, method_name, outbound, outbound_size);
- ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
+ osd_req_op_cls_response_data_pages(obj_request->osd_req, 0,
+ obj_request->pages, 0, 0,
false, false);
- osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
rbd_osd_req_format(obj_request, false);

ret = rbd_obj_request_submit(osdc, obj_request);
@@ -2037,7 +2031,6 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
{
struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc; struct rbd_obj_request *obj_request;
- struct ceph_osd_data *osd_data;
struct page **pages = NULL;
u32 page_count;
size_t size;
@@ -2061,14 +2054,13 @@ static int rbd_obj_read_sync(struct rbd_device *rbd_dev,
if (!obj_request->osd_req)
goto out;

- osd_data = &obj_request->osd_req->r_data_in;
osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
offset, length, 0, 0);
- ceph_osd_data_pages_init(osd_data, obj_request->pages,
+ osd_req_op_extent_osd_data_pages(obj_request->osd_req, 0, false,
+ obj_request->pages,
obj_request->length, obj_request->offset & ~PAGE_MASK, false, false);
- osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
rbd_osd_req_format(obj_request, false);

ret = rbd_obj_request_submit(osdc, obj_request);

fs/ceph/addr.c +0 -0
index dd5d263..068d2c8 100644
@@ -245,7 +245,7 @@ static void finish_read(struct ceph_osd_request

*req, struct ceph_msg *msg)
dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);

/* unlock all pages, zeroing any data we didn't read */
- osd_data = &req->r_data_in;
+ osd_data = osd_req_op_extent_osd_data(req, 0, false);
BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES); num_pages = calc_pages_for((u64)osd_data->alignment,
(u64)osd_data->length);
@@ -343,8 +343,7 @@ static int start_read(struct inode *inode, struct list_head *page_list, int max)
}
pages[i] = page;
}
- BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
- ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
+ osd_req_op_extent_osd_data_pages(req, 0, false, pages, len, 0,
false, false);
req->r_callback = finish_read;
req->r_inode = inode;
@@ -572,7 +571,7 @@ static void writepages_finish(struct ceph_osd_request *req,
long writeback_stat;
unsigned issued = ceph_caps_issued(ci);

- osd_data = &req->r_data_out;
+ osd_data = osd_req_op_extent_osd_data(req, 0, true);
BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES); num_pages = calc_pages_for((u64)osd_data->alignment,
(u64)osd_data->length);
@@ -917,9 +916,8 @@ get_more_pages:
dout("writepages got %d pages at %llu~%llu\n",
locked_pages, offset, len);

- BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
- ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
- len, 0, !!pool, false);
+ osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, 0,
+ !!pool, false);

pages = NULL; /* request message now owns the pages array */
pool = NULL;

fs/ceph/file.c +1 -2
index 66d0938..b7e6caa 100644
@@ -574,8 +574,7 @@ more:
 			own_pages = true;
 		}
 	}
-	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
-	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
+	osd_req_op_extent_osd_data_pages(req, 0, true, pages, len,
 					page_align, false, own_pages);
 
 	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index 71c4157..f8a00b4 100644

include/linux/ceph/osd_client.h +0 -0
@@ -240,17 +240,39 @@ extern void osd_req_op_extent_init(struct

ceph_osd_request *osd_req,
u64 truncate_size, u32 truncate_seq);
extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
unsigned int which, u64 length);
-extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req, +
+extern struct ceph_osd_data *osd_req_op_extent_osd_data(
+ struct ceph_osd_request *osd_req,
+ unsigned int which, bool write_request);
+extern struct ceph_osd_data *osd_req_op_cls_response_data(
+ struct ceph_osd_request *osd_req,
+ unsigned int which);
+
+extern void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *,
+ unsigned int which, bool write_request,
+ struct page **pages, u64 length,
+ u32 alignment, bool pages_from_pool,
+ bool own_pages);
+extern void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *,
+ unsigned int which, bool write_request,
+ struct ceph_pagelist *pagelist);
+#ifdef CONFIG_BLOCK
+extern void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *,
+ unsigned int which, bool write_request,
+ struct bio *bio, size_t bio_length);
+#endif /* CONFIG_BLOCK */
+
+extern void osd_req_op_cls_response_data_pages(struct ceph_osd_request *,
unsigned int which,
- struct ceph_osd_data *osd_data);
+ struct page **pages, u64 length,
+ u32 alignment, bool pages_from_pool,
+ bool own_pages);
+
extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
unsigned int which, u16 opcode, const char *class, const char *method, const void *request_data, size_t request_data_size);
-extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
- unsigned int which,
- struct ceph_osd_data *response_data);
extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
unsigned int which, u16 opcode, u64 cookie, u64 version, int flag);
@@ -290,17 +312,6 @@ static inline void ceph_osdc_put_request(struct ceph_osd_request *req)
kref_put(&req->r_kref, ceph_osdc_release_request);
}

-extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,

-				     struct page **pages, u64 length,
-				     u32 alignment, bool pages_from_pool,
-				     bool own_pages);

-extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
- struct ceph_pagelist *pagelist);
-#ifdef CONFIG_BLOCK
-extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
- struct bio *bio, size_t bio_length);
-#endif /* CONFIG_BLOCK */
-
extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
struct ceph_osd_request *req, bool nofail);

net/ceph/osd_client.c +1 -0
index 40466ab..86cb524 100644
@@ -1,3 +1,4 @@
+
 #include <linux/ceph/ceph_debug.h>
 
 #include <linux/module.h>
@@ -81,11 +82,11 @@ static int calc_layout(struct ceph_file_layout

*layout, u64 off, u64 *plen,

static void ceph_osd_data_init(struct ceph_osd_data *osd_data) {
- memset(osd_data, 0, sizeof *osd_data);
+ memset(osd_data, 0, sizeof (*osd_data));
osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
}

-void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data, +static void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
struct page **pages, u64 length, u32 alignment, bool pages_from_pool, bool own_pages)
{
@@ -96,27 +97,131 @@ void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
osd_data->pages_from_pool = pages_from_pool; osd_data->own_pages = own_pages;
}
-EXPORT_SYMBOL(ceph_osd_data_pages_init);

-void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data, +static void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
struct ceph_pagelist *pagelist)
{
osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST; osd_data->pagelist = pagelist;
}
-EXPORT_SYMBOL(ceph_osd_data_pagelist_init);

#ifdef CONFIG_BLOCK
-void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data, +static void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
struct bio *bio, size_t bio_length)
{
osd_data->type = CEPH_OSD_DATA_TYPE_BIO; osd_data->bio = bio;
osd_data->bio_length = bio_length;
}
-EXPORT_SYMBOL(ceph_osd_data_bio_init);
#endif /* CONFIG_BLOCK */

+struct ceph_osd_data *
+osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+ unsigned int which, bool write_request)
+{
+ BUG_ON(which >= osd_req->r_num_ops);
+
+ /* return &osd_req->r_ops[which].extent.osd_data; */
+ return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data);
+
+struct ceph_osd_data *
+osd_req_op_cls_request_info(struct ceph_osd_request *osd_req,
+ unsigned int which)
+{
+ BUG_ON(which >= osd_req->r_num_ops);
+
+ /* return &osd_req->r_ops[which].cls.request_info; */
+ return &osd_req->r_data_out; /* Request data is outgoing */
+}
+EXPORT_SYMBOL(osd_req_op_cls_request_info); /* ??? */
+
+struct ceph_osd_data *
+osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+ unsigned int which)
+{
+ BUG_ON(which >= osd_req->r_num_ops);
+
+ /* return &osd_req->r_ops[which].cls.response_data; */
+ return &osd_req->r_data_in; /* Response data is incoming */
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data); /* ??? */
+
+void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *osd_req,
+ unsigned int which, bool write_request,
+ struct page **pages, u64 length, u32 alignment,
+ bool pages_from_pool, bool own_pages)
+{
+ struct ceph_osd_data *osd_data;
+
+ osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+ ceph_osd_data_pages_init(osd_data, pages, length, alignment,
+ pages_from_pool, own_pages);
+
+ osd_req->r_ops[which].extent.osd_data =
+ osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);
+
+void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *osd_req,
+ unsigned int which, bool write_request,
+ struct ceph_pagelist *pagelist)
+{
+ struct ceph_osd_data *osd_data;
+
+ osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+ ceph_osd_data_pagelist_init(osd_data, pagelist);
+
+ osd_req->r_ops[which].extent.osd_data =
+ osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);
+
+#ifdef CONFIG_BLOCK
+void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *osd_req,
+ unsigned int which, bool write_request,
+ struct bio *bio, size_t bio_length)
+{
+ struct ceph_osd_data *osd_data;
+
+ osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+ ceph_osd_data_bio_init(osd_data, bio, bio_length);
+
+ osd_req->r_ops[which].extent.osd_data =
+ osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
+#endif /* CONFIG_BLOCK */
+
+static void osd_req_op_cls_request_info_pagelist(
+ struct ceph_osd_request *osd_req,
+ unsigned int which, struct ceph_pagelist *pagelist)
+{
+ struct ceph_osd_data *osd_data;
+
+ osd_data = osd_req_op_cls_request_info(osd_req, which);
+ ceph_osd_data_pagelist_init(osd_data, pagelist);
+
+ osd_req->r_ops[which].cls.request_info =
+ osd_req_op_cls_request_info(osd_req, which);
+}
+
+void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
+ unsigned int which, struct page **pages, u64 length,
+ u32 alignment, bool pages_from_pool, bool own_pages)
+{
+ struct ceph_osd_data *osd_data;
+
+ osd_data = osd_req_op_cls_response_data(osd_req, which);
+ ceph_osd_data_pages_init(osd_data, pages, length, alignment,
+ pages_from_pool, own_pages);
+
+ osd_req->r_ops[which].cls.response_data =
+ osd_req_op_cls_response_data(osd_req, which);
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);
+
static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data) {
switch (osd_data->type) {
@@ -385,15 +490,6 @@ void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
}
EXPORT_SYMBOL(osd_req_op_extent_update);

-void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
- unsigned int which,
- struct ceph_osd_data *osd_data)
-{
- BUG_ON(which >= osd_req->r_num_ops);
- osd_req->r_ops[which].extent.osd_data = osd_data;
-}
-EXPORT_SYMBOL(osd_req_op_extent_osd_data);
-
void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int
which,
u16 opcode, const char *class, const char *method, const void *request_data, size_t request_data_size)
@@ -429,22 +525,13 @@ void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int which,
ceph_pagelist_append(pagelist, request_data, request_data_size); payload_len += request_data_size;

- op->cls.request_info = &osd_req->r_data_out;
- ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
+ osd_req_op_cls_request_info_pagelist(osd_req, which, pagelist);

op->cls.argc = 0; /* currently unused */

op->payload_len = payload_len;
}
EXPORT_SYMBOL(osd_req_op_cls_init);
-void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
- unsigned int which,
- struct ceph_osd_data *response_data)
-{
- BUG_ON(which >= osd_req->r_num_ops);
- osd_req->r_ops[which].cls.response_data = response_data;
-}
-EXPORT_SYMBOL(osd_req_op_cls_response_data);

void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
unsigned int which, u16 opcode,
@@ -547,7 +634,6 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
bool use_mempool)
{
struct ceph_osd_request *req;
- struct ceph_osd_data *osd_data;
u64 objnum = 0;
u64 objoff = 0;
u64 objlen = 0;
@@ -561,8 +647,6 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,
GFP_NOFS);
if (!req)
return ERR_PTR(-ENOMEM);
- osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
- : &req->r_data_in;

req->r_flags = flags;

@@ -585,7 +669,6 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc,

osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
truncate_size, truncate_seq);
- osd_req_op_extent_osd_data(req, 0, osd_data);

/*
* A second op in the ops array means the caller wants to
@@ -2171,8 +2254,8 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,

/* it may be a short read due to an object boundary */

- ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
- false, false);
+ osd_req_op_extent_osd_data_pages(req, 0, false,
+ pages, *plen, page_align, false, false);

dout("readpages final extent is %llu~%llu (%llu bytes align %d)\n",
off, *plen, *plen, page_align);
@@ -2214,7 +2297,7 @@ int ceph_osdc_writepages(struct ceph_osd_client *osdc, struct ceph_vino vino,
return PTR_ERR(req);

/* it may be a short write due to an object boundary */
- ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+ osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, page_align,
false, false);
dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);

@@ -2308,8 +2391,14 @@ static struct ceph_msg *get_reply(struct ceph_connection *con,
m = ceph_msg_get(req->r_reply);

if (data_len > 0) {
- struct ceph_osd_data *osd_data = &req->r_data_in;
+ struct ceph_osd_data *osd_data;

+ /*
+ * XXX This is assuming there is only one op containing
+ * XXX page data. Probably OK for reads, but this
+ * XXX ought to be done more generally.
+ */
+ osd_data = osd_req_op_extent_osd_data(req, 0, false);
if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
if (osd_data->pages &&
unlikely(osd_data->length < data_len)) {

Signature · 1 line
1.7.9.5
Reviewed-by on v1 18/20
#36 lore raw

Re: [PATCH 18/20] libceph: combine initializing and setting osd data

Quoted 521 lines · Alex Elder

This ends up being a rather large patch but what it's doing is somewhat straightforward.

Basically, this is replacing two calls with one. The first of the two calls is initializing a struct ceph_osd_data with data (either a page array, a page list, or a bio list); the second is setting an osd request op so it associates that data with one of the op's parameters. In place of those two will be a single function that initializes the op directly.

That means we sort of fan out a set of the needed functions:
- extent ops with pages data
- extent ops with pagelist data
- extent ops with bio list data
and
- class ops with page data for receiving a response

We also have define another one, but it's only used internally:
- class ops with pagelist data for request parameters

Note that we *still* haven't gotten rid of the osd request's r_data_in and r_data_out fields. All the osd ops refer to them for their data. For now, these data fields are pointers assigned to the appropriate r_data_* field when these new functions are called.

---
  drivers/block/rbd.c             |   20 ++---
  fs/ceph/addr.c                  |   12 ++-
  fs/ceph/file.c                  |    3 +-
  include/linux/ceph/osd_client.h |   43 +++++++----
  net/ceph/osd_client.c           |  157
++++++++++++++++++++++++++++++---------
  5 files changed, 162 insertions(+), 73 deletions(-)
drivers/block/rbd.c +0 -1
index 3e8e6d5..7db244e 100644
@@ -1592,7 +1592,6 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  	rbd_assert(resid > 0);
  	while (resid) {
  		struct ceph_osd_request *osd_req;
-		struct ceph_osd_data *osd_data;
  		const char *object_name;
  		unsigned int clone_size;

u64 offset;

drivers/block/rbd.c +2 -5
@@ -1625,13 +1624,10 @@ static int rbd_img_request_fill_bio(struct
 rbd_img_request *img_request,
  		obj_request->osd_req = osd_req;
  		obj_request->callback = rbd_img_obj_callback;
 
-		osd_data = write_request ? &osd_req->r_data_out
-					 : &osd_req->r_data_in;
  		osd_req_op_extent_init(osd_req, 0, opcode, offset, length,
  						0, 0);
-		ceph_osd_data_bio_init(osd_data, obj_request->bio_list,
-					obj_request->length);
-		osd_req_op_extent_osd_data(osd_req, 0, osd_data);
+		osd_req_op_extent_osd_data_bio(osd_req, 0, write_request,
+				obj_request->bio_list, obj_request->length);
  		rbd_osd_req_format(obj_request, write_request);
 
  		rbd_img_obj_request_add(img_request, obj_request);
drivers/block/rbd.c +0 -1
@@ -1821,7 +1817,6 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  {
  	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
  	struct rbd_obj_request *obj_request;
-	struct ceph_osd_data *osd_data;
  	struct page **pages;
  	u32 page_count;

int ret;

drivers/block/rbd.c +2 -3
@@ -1851,13 +1846,12 @@ static int rbd_obj_method_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	osd_data = &obj_request->osd_req->r_data_in;
  	osd_req_op_cls_init(obj_request->osd_req, 0, CEPH_OSD_OP_CALL,
  					class_name, method_name,
  					outbound, outbound_size);
-	ceph_osd_data_pages_init(osd_data, obj_request->pages, 0, 0,
+	osd_req_op_cls_response_data_pages(obj_request->osd_req, 0,
+					obj_request->pages, 0, 0,
  					false, false);
-	osd_req_op_cls_response_data(obj_request->osd_req, 0, osd_data);
  	rbd_osd_req_format(obj_request, false);
 
  	ret = rbd_obj_request_submit(osdc, obj_request);
drivers/block/rbd.c +0 -1
@@ -2037,7 +2031,6 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  {
  	struct ceph_osd_client *osdc = &rbd_dev->rbd_client->client->osdc;
  	struct rbd_obj_request *obj_request;
-	struct ceph_osd_data *osd_data;
  	struct page **pages = NULL;
  	u32 page_count;

size_t size;

drivers/block/rbd.c +2 -3
@@ -2061,14 +2054,13 @@ static int rbd_obj_read_sync(struct rbd_device
 *rbd_dev,
  	if (!obj_request->osd_req)
  		goto out;
 
-	osd_data = &obj_request->osd_req->r_data_in;
  	osd_req_op_extent_init(obj_request->osd_req, 0, CEPH_OSD_OP_READ,
  					offset, length, 0, 0);
-	ceph_osd_data_pages_init(osd_data, obj_request->pages,
+	osd_req_op_extent_osd_data_pages(obj_request->osd_req, 0, false,
+					obj_request->pages,
  					obj_request->length,
  					obj_request->offset & ~PAGE_MASK,
  					false, false);
-	osd_req_op_extent_osd_data(obj_request->osd_req, 0, osd_data);
  	rbd_osd_req_format(obj_request, false);
 
  	ret = rbd_obj_request_submit(osdc, obj_request);
fs/ceph/addr.c +1 -1
index dd5d263..068d2c8 100644
@@ -245,7 +245,7 @@ static void finish_read(struct ceph_osd_request
 *req, struct ceph_msg *msg)
  	dout("finish_read %p req %p rc %d bytes %d\n", inode, req, rc, bytes);
 
  	/* unlock all pages, zeroing any data we didn't read */
-	osd_data = &req->r_data_in;
+	osd_data = osd_req_op_extent_osd_data(req, 0, false);
  	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
  	num_pages = calc_pages_for((u64)osd_data->alignment,
  					(u64)osd_data->length);
fs/ceph/addr.c +1 -2
@@ -343,8 +343,7 @@ static int start_read(struct inode *inode, struct
 list_head *page_list, int max)
  		}
  		pages[i] = page;
  	}
-	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_in);
-	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len, 0,
+	osd_req_op_extent_osd_data_pages(req, 0, false, pages, len, 0,
  					false, false);
  	req->r_callback = finish_read;
  	req->r_inode = inode;
fs/ceph/addr.c +1 -1
@@ -572,7 +571,7 @@ static void writepages_finish(struct
 ceph_osd_request *req,
  	long writeback_stat;
  	unsigned issued = ceph_caps_issued(ci);
 
-	osd_data = &req->r_data_out;
+	osd_data = osd_req_op_extent_osd_data(req, 0, true);
  	BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_PAGES);
  	num_pages = calc_pages_for((u64)osd_data->alignment,
  					(u64)osd_data->length);
fs/ceph/addr.c +2 -3
@@ -917,9 +916,8 @@ get_more_pages:
  		dout("writepages got %d pages at %llu~%llu\n",
  		     locked_pages, offset, len);
 
-		BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
-		ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages,
-						len, 0, !!pool, false);
+		osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, 0,
+							!!pool, false);
 
  		pages = NULL;	/* request message now owns the pages array */
  		pool = NULL;
fs/ceph/file.c +1 -2
index 66d0938..b7e6caa 100644
@@ -574,8 +574,7 @@ more:
  			own_pages = true;
  		}
  	}
-	BUG_ON(req->r_ops[0].extent.osd_data != &req->r_data_out);
-	ceph_osd_data_pages_init(req->r_ops[0].extent.osd_data, pages, len,
+	osd_req_op_extent_osd_data_pages(req, 0, true, pages, len,
  					page_align, false, own_pages);
 
  	/* BUG_ON(vino.snap != CEPH_NOSNAP); */
include/linux/ceph/osd_client.h +27 -5
index 71c4157..f8a00b4 100644
@@ -240,17 +240,39 @@ extern void osd_req_op_extent_init(struct
 ceph_osd_request *osd_req,
  					u64 truncate_size, u32 truncate_seq);
  extern void osd_req_op_extent_update(struct ceph_osd_request *osd_req,
  					unsigned int which, u64 length);
-extern void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+
+extern struct ceph_osd_data *osd_req_op_extent_osd_data(
+					struct ceph_osd_request *osd_req,
+					unsigned int which, bool write_request);
+extern struct ceph_osd_data *osd_req_op_cls_response_data(
+					struct ceph_osd_request *osd_req,
+					unsigned int which);
+
+extern void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *,
+					unsigned int which, bool write_request,
+					struct page **pages, u64 length,
+					u32 alignment, bool pages_from_pool,
+					bool own_pages);
+extern void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *,
+					unsigned int which, bool write_request,
+					struct ceph_pagelist *pagelist);
+#ifdef CONFIG_BLOCK
+extern void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *,
+					unsigned int which, bool write_request,
+					struct bio *bio, size_t bio_length);
+#endif /* CONFIG_BLOCK */
+
+extern void osd_req_op_cls_response_data_pages(struct ceph_osd_request *,
  					unsigned int which,
-					struct ceph_osd_data *osd_data);
+					struct page **pages, u64 length,
+					u32 alignment, bool pages_from_pool,
+					bool own_pages);
+
  extern void osd_req_op_cls_init(struct ceph_osd_request *osd_req,
  					unsigned int which, u16 opcode,
  					const char *class, const char *method,
  					const void *request_data,
  					size_t request_data_size);
-extern void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
-					unsigned int which,
-					struct ceph_osd_data *response_data);
  extern void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
  					unsigned int which, u16 opcode,

u64 cookie, u64 version, int flag);

include/linux/ceph/osd_client.h +0 -11
@@ -290,17 +312,6 @@ static inline void ceph_osdc_put_request(struct
 ceph_osd_request *req)
  	kref_put(&req->r_kref, ceph_osdc_release_request);
  }
 
-extern void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
-				     struct page **pages, u64 length,
-				     u32 alignment, bool pages_from_pool,
-				     bool own_pages);
-extern void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
-					struct ceph_pagelist *pagelist);
-#ifdef CONFIG_BLOCK
-extern void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
-				   struct bio *bio, size_t bio_length);
-#endif /* CONFIG_BLOCK */
-
  extern int ceph_osdc_start_request(struct ceph_osd_client *osdc,
  				   struct ceph_osd_request *req,

bool nofail);

net/ceph/osd_client.c +3 -2
index 40466ab..86cb524 100644
@@ -1,3 +1,4 @@
+
  #include <linux/ceph/ceph_debug.h>
 
  #include <linux/module.h>
@@ -81,11 +82,11 @@ static int calc_layout(struct ceph_file_layout
 *layout, u64 off, u64 *plen,
 
  static void ceph_osd_data_init(struct ceph_osd_data *osd_data)
  {
-	memset(osd_data, 0, sizeof *osd_data);
+	memset(osd_data, 0, sizeof (*osd_data));
  	osd_data->type = CEPH_OSD_DATA_TYPE_NONE;
  }
 
-void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
+static void ceph_osd_data_pages_init(struct ceph_osd_data *osd_data,
  			struct page **pages, u64 length, u32 alignment,
  			bool pages_from_pool, bool own_pages)
  {
net/ceph/osd_client.c +109 -5
@@ -96,27 +97,131 @@ void ceph_osd_data_pages_init(struct ceph_osd_data
 *osd_data,
  	osd_data->pages_from_pool = pages_from_pool;
  	osd_data->own_pages = own_pages;
  }
-EXPORT_SYMBOL(ceph_osd_data_pages_init);
 
-void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
+static void ceph_osd_data_pagelist_init(struct ceph_osd_data *osd_data,
  			struct ceph_pagelist *pagelist)
  {
  	osd_data->type = CEPH_OSD_DATA_TYPE_PAGELIST;
  	osd_data->pagelist = pagelist;
  }
-EXPORT_SYMBOL(ceph_osd_data_pagelist_init);
 
  #ifdef CONFIG_BLOCK
-void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
+static void ceph_osd_data_bio_init(struct ceph_osd_data *osd_data,
  			struct bio *bio, size_t bio_length)
  {
  	osd_data->type = CEPH_OSD_DATA_TYPE_BIO;
  	osd_data->bio = bio;
  	osd_data->bio_length = bio_length;
  }
-EXPORT_SYMBOL(ceph_osd_data_bio_init);
  #endif /* CONFIG_BLOCK */
 
+struct ceph_osd_data *
+osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request)
+{
+	BUG_ON(which >= osd_req->r_num_ops);
+
+	/* return &osd_req->r_ops[which].extent.osd_data; */
+	return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data);
+
+struct ceph_osd_data *
+osd_req_op_cls_request_info(struct ceph_osd_request *osd_req,
+			unsigned int which)
+{
+	BUG_ON(which >= osd_req->r_num_ops);
+
+	/* return &osd_req->r_ops[which].cls.request_info; */
+	return &osd_req->r_data_out;	/* Request data is outgoing */
+}
+EXPORT_SYMBOL(osd_req_op_cls_request_info);	/* ??? */
+
+struct ceph_osd_data *
+osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
+			unsigned int which)
+{
+	BUG_ON(which >= osd_req->r_num_ops);
+
+	/* return &osd_req->r_ops[which].cls.response_data; */
+	return &osd_req->r_data_in;	/* Response data is incoming */
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data);	/* ??? */
+
+void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request,
+			struct page **pages, u64 length, u32 alignment,
+			bool pages_from_pool, bool own_pages)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
+				pages_from_pool, own_pages);
+
+	osd_req->r_ops[which].extent.osd_data =
+		osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);
+
+void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request,
+			struct ceph_pagelist *pagelist)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+	ceph_osd_data_pagelist_init(osd_data, pagelist);
+
+	osd_req->r_ops[which].extent.osd_data =
+		osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);
+
+#ifdef CONFIG_BLOCK
+void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *osd_req,
+			unsigned int which, bool write_request,
+			struct bio *bio, size_t bio_length)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
+	ceph_osd_data_bio_init(osd_data, bio, bio_length);
+
+	osd_req->r_ops[which].extent.osd_data =
+		osd_req_op_extent_osd_data(osd_req, which, write_request);
+}
+EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
+#endif /* CONFIG_BLOCK */
+
+static void osd_req_op_cls_request_info_pagelist(
+			struct ceph_osd_request *osd_req,
+			unsigned int which, struct ceph_pagelist *pagelist)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_cls_request_info(osd_req, which);
+	ceph_osd_data_pagelist_init(osd_data, pagelist);
+
+	osd_req->r_ops[which].cls.request_info =
+		osd_req_op_cls_request_info(osd_req, which);
+}
+
+void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
+			unsigned int which, struct page **pages, u64 length,
+			u32 alignment, bool pages_from_pool, bool own_pages)
+{
+	struct ceph_osd_data *osd_data;
+
+	osd_data = osd_req_op_cls_response_data(osd_req, which);
+	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
+				pages_from_pool, own_pages);
+
+	osd_req->r_ops[which].cls.response_data =
+		osd_req_op_cls_response_data(osd_req, which);
+}
+EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);
+
  static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data)
  {
  	switch (osd_data->type) {
net/ceph/osd_client.c +0 -9
@@ -385,15 +490,6 @@ void osd_req_op_extent_update(struct
 ceph_osd_request *osd_req,
  }
  EXPORT_SYMBOL(osd_req_op_extent_update);
 
-void osd_req_op_extent_osd_data(struct ceph_osd_request *osd_req,
-				unsigned int which,
-				struct ceph_osd_data *osd_data)
-{
-	BUG_ON(which >= osd_req->r_num_ops);
-	osd_req->r_ops[which].extent.osd_data = osd_data;
-}
-EXPORT_SYMBOL(osd_req_op_extent_osd_data);
-
  void osd_req_op_cls_init(struct ceph_osd_request *osd_req, unsigned int
 which,

u16 opcode, const char *class, const char *method, const void *request_data, size_t request_data_size)

net/ceph/osd_client.c +1 -10
@@ -429,22 +525,13 @@ void osd_req_op_cls_init(struct ceph_osd_request
 *osd_req, unsigned int which,
  	ceph_pagelist_append(pagelist, request_data, request_data_size);
  	payload_len += request_data_size;
 
-	op->cls.request_info = &osd_req->r_data_out;
-	ceph_osd_data_pagelist_init(op->cls.request_info, pagelist);
+	osd_req_op_cls_request_info_pagelist(osd_req, which, pagelist);
 
  	op->cls.argc = 0;	/* currently unused */
 
  	op->payload_len = payload_len;
  }
  EXPORT_SYMBOL(osd_req_op_cls_init);
-void osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
-				unsigned int which,
-				struct ceph_osd_data *response_data)
-{
-	BUG_ON(which >= osd_req->r_num_ops);
-	osd_req->r_ops[which].cls.response_data = response_data;
-}
-EXPORT_SYMBOL(osd_req_op_cls_response_data);
 
  void osd_req_op_watch_init(struct ceph_osd_request *osd_req,

unsigned int which, u16 opcode,

net/ceph/osd_client.c +0 -1
@@ -547,7 +634,6 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  					       bool use_mempool)
  {
  	struct ceph_osd_request *req;
-	struct ceph_osd_data *osd_data;
  	u64 objnum = 0;
  	u64 objoff = 0;
  	u64 objlen = 0;
net/ceph/osd_client.c +0 -3
@@ -561,8 +647,6 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
  					GFP_NOFS);
  	if (!req)
  		return ERR_PTR(-ENOMEM);
-	osd_data = opcode == CEPH_OSD_OP_WRITE ? &req->r_data_out
-					       : &req->r_data_in;
 
  	req->r_flags = flags;
@@ -585,7 +669,6 @@ struct ceph_osd_request
 *ceph_osdc_new_request(struct ceph_osd_client *osdc,
 
  	osd_req_op_extent_init(req, 0, opcode, objoff, objlen,
  				truncate_size, truncate_seq);
-	osd_req_op_extent_osd_data(req, 0, osd_data);
 
  	/*

* A second op in the ops array means the caller wants to

net/ceph/osd_client.c +10 -4
@@ -2171,8 +2254,8 @@ int ceph_osdc_readpages(struct ceph_osd_client *osdc,
 
  	/* it may be a short read due to an object boundary */
 
-	ceph_osd_data_pages_init(&req->r_data_in, pages, *plen, page_align,
-				false, false);
+	osd_req_op_extent_osd_data_pages(req, 0, false,
+				pages, *plen, page_align, false, false);
 
  	dout("readpages  final extent is %llu~%llu (%llu bytes align %d)\n",
  	     off, *plen, *plen, page_align);
@@ -2214,7 +2297,7 @@ int ceph_osdc_writepages(struct ceph_osd_client
 *osdc, struct ceph_vino vino,
  		return PTR_ERR(req);
 
  	/* it may be a short write due to an object boundary */
-	ceph_osd_data_pages_init(&req->r_data_out, pages, len, page_align,
+	osd_req_op_extent_osd_data_pages(req, 0, true, pages, len, page_align,
  				false, false);
  	dout("writepages %llu~%llu (%llu bytes)\n", off, len, len);
@@ -2308,8 +2391,14 @@ static struct ceph_msg *get_reply(struct
 ceph_connection *con,
  	m = ceph_msg_get(req->r_reply);
 
  	if (data_len > 0) {
-		struct ceph_osd_data *osd_data = &req->r_data_in;
+		struct ceph_osd_data *osd_data;
 
+		/*
+		 * XXX This is assuming there is only one op containing
+		 * XXX page data.  Probably OK for reads, but this
+		 * XXX ought to be done more generally.
+		 */
+		osd_data = osd_req_op_extent_osd_data(req, 0, false);
  		if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
  			if (osd_data->pages &&
  				unlikely(osd_data->length < data_len)) {
author
#37 lore raw

PATCH 19/20 libceph: set the data pointers when encoding ops

In reply to Alex Elder

Still using the osd request r_data_in and r_data_out pointer, but we're basically only referring to it via the data pointers in the osd ops. And we're transferring that information to the request or reply message only when the op indicates it's needed, in osd_req_encode_op().

To avoid a forward reference, ceph_osdc_msg_data_set() was moved up in the file.

Don't bother calling ceph_osd_data_init(), in ceph_osd_alloc(), because the ops array will already be zeroed anyway.

---
 include/linux/ceph/osd_client.h |    2 +-
 net/ceph/osd_client.c           |   63
+++++++++++++++++++--------------------
 2 files changed, 32 insertions(+), 33 deletions(-)
(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index f8a00b4..dd4ca4ba 100644

include/linux/ceph/osd_client.h +1 -1
@@ -51,7 +51,7 @@ struct ceph_osd {
 #define CEPH_OSD_MAX_OP	2
 
 enum ceph_osd_data_type {
-	CEPH_OSD_DATA_TYPE_NONE,
+	CEPH_OSD_DATA_TYPE_NONE = 0,
 	CEPH_OSD_DATA_TYPE_PAGES,
 	CEPH_OSD_DATA_TYPE_PAGELIST,
 #ifdef CONFIG_BLOCK
net/ceph/osd_client.c +0 -0
index 86cb524..cc4003f 100644
@@ -339,9 +339,6 @@ struct ceph_osd_request

*ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
}
req->r_reply = msg;

- ceph_osd_data_init(&req->r_data_in);
- ceph_osd_data_init(&req->r_data_out);
-
/* create request message; allow space for oid */
if (use_mempool)
msg = ceph_msgpool_get(&osdc->msgpool_op, 0);
@@ -549,6 +546,28 @@ void osd_req_op_watch_init(struct ceph_osd_request *osd_req,
}
EXPORT_SYMBOL(osd_req_op_watch_init);

+static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
+ struct ceph_osd_data *osd_data)
+{
+ u64 length = ceph_osd_data_length(osd_data);
+
+ if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
+ BUG_ON(length > (u64) SIZE_MAX);
+ if (length)
+ ceph_msg_data_set_pages(msg, osd_data->pages,
+ length, osd_data->alignment);
+ } else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
+ BUG_ON(!length);
+ ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
+#ifdef CONFIG_BLOCK
+ } else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
+ ceph_msg_data_set_bio(msg, osd_data->bio, length);
+#endif
+ } else {
+ BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
+ }
+}
+
static u64 osd_req_encode_op(struct ceph_osd_request *req,
struct ceph_osd_op *dst, unsigned int which)
{
@@ -576,17 +595,24 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
cpu_to_le64(src->extent.truncate_size);
dst->extent.truncate_seq =
cpu_to_le32(src->extent.truncate_seq);
- if (src->op == CEPH_OSD_OP_WRITE)
+ if (src->op == CEPH_OSD_OP_WRITE) {
WARN_ON(src->extent.osd_data != &req->r_data_out);
- else
+ ceph_osdc_msg_data_set(req->r_request,
+ src->extent.osd_data);
+ } else {
WARN_ON(src->extent.osd_data != &req->r_data_in);
+ ceph_osdc_msg_data_set(req->r_reply,
+ src->extent.osd_data);
+ }
break;
case CEPH_OSD_OP_CALL:
dst->cls.class_len = src->cls.class_len; dst->cls.method_len = src->cls.method_len; dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len); WARN_ON(src->cls.response_data != &req->r_data_in);
+ ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
WARN_ON(src->cls.request_info != &req->r_data_out);
+ ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
BUG_ON(src->cls.request_info->type !=
CEPH_OSD_DATA_TYPE_PAGELIST);
request_data_len = src->cls.request_info->pagelist->length;
@@ -1930,28 +1956,6 @@ bad:
return;
}

-static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
- struct ceph_osd_data *osd_data)
-{
- u64 length = ceph_osd_data_length(osd_data);
-
- if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
- BUG_ON(length > (u64) SIZE_MAX);
- if (length)
- ceph_msg_data_set_pages(msg, osd_data->pages,
- length, osd_data->alignment);
- } else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
- BUG_ON(!length);
- ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
-#ifdef CONFIG_BLOCK
- } else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
- ceph_msg_data_set_bio(msg, osd_data->bio, length);
-#endif
- } else {
- BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
- }
-}
-
/*
* build new request AND message
*
@@ -1967,11 +1971,6 @@ void ceph_osdc_build_request(struct ceph_osd_request *req, u64 off,
u64 data_len;
unsigned int i;

- /* Set up response incoming data and request outgoing data fields */
-
- ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
- ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
-
req->r_snapid = snap_id; req->r_snapc = ceph_get_snap_context(snapc);

Signature · 1 line
1.7.9.5
Reviewed-by on v1 19/20
#38 lore raw

Re: [PATCH 19/20] libceph: set the data pointers when encoding ops

Quoted 148 lines · Alex Elder

Still using the osd request r_data_in and r_data_out pointer, but we're basically only referring to it via the data pointers in the osd ops. And we're transferring that information to the request or reply message only when the op indicates it's needed, in osd_req_encode_op().

To avoid a forward reference, ceph_osdc_msg_data_set() was moved up in the file.

Don't bother calling ceph_osd_data_init(), in ceph_osd_alloc(), because the ops array will already be zeroed anyway.

---
  include/linux/ceph/osd_client.h |    2 +-
  net/ceph/osd_client.c           |   63
+++++++++++++++++++--------------------
  2 files changed, 32 insertions(+), 33 deletions(-)
include/linux/ceph/osd_client.h +1 -1
index f8a00b4..dd4ca4ba 100644
@@ -51,7 +51,7 @@ struct ceph_osd {
  #define CEPH_OSD_MAX_OP	2
 
  enum ceph_osd_data_type {
-	CEPH_OSD_DATA_TYPE_NONE,
+	CEPH_OSD_DATA_TYPE_NONE = 0,
  	CEPH_OSD_DATA_TYPE_PAGES,
  	CEPH_OSD_DATA_TYPE_PAGELIST,
  #ifdef CONFIG_BLOCK
net/ceph/osd_client.c +0 -3
index 86cb524..cc4003f 100644
@@ -339,9 +339,6 @@ struct ceph_osd_request
 *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
  	}
  	req->r_reply = msg;
 
-	ceph_osd_data_init(&req->r_data_in);
-	ceph_osd_data_init(&req->r_data_out);
-
  	/* create request message; allow space for oid */
  	if (use_mempool)
  		msg = ceph_msgpool_get(&osdc->msgpool_op, 0);
net/ceph/osd_client.c +22 -0
@@ -549,6 +546,28 @@ void osd_req_op_watch_init(struct ceph_osd_request
 *osd_req,
  }
  EXPORT_SYMBOL(osd_req_op_watch_init);
 
+static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
+				struct ceph_osd_data *osd_data)
+{
+	u64 length = ceph_osd_data_length(osd_data);
+
+	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
+		BUG_ON(length > (u64) SIZE_MAX);
+		if (length)
+			ceph_msg_data_set_pages(msg, osd_data->pages,
+					length, osd_data->alignment);
+	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
+		BUG_ON(!length);
+		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
+#ifdef CONFIG_BLOCK
+	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
+		ceph_msg_data_set_bio(msg, osd_data->bio, length);
+#endif
+	} else {
+		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
+	}
+}
+
  static u64 osd_req_encode_op(struct ceph_osd_request *req,
  			      struct ceph_osd_op *dst, unsigned int which)
  {
net/ceph/osd_client.c +9 -2
@@ -576,17 +595,24 @@ static u64 osd_req_encode_op(struct
 ceph_osd_request *req,
  			cpu_to_le64(src->extent.truncate_size);
  		dst->extent.truncate_seq =
  			cpu_to_le32(src->extent.truncate_seq);
-		if (src->op == CEPH_OSD_OP_WRITE)
+		if (src->op == CEPH_OSD_OP_WRITE) {
  			WARN_ON(src->extent.osd_data != &req->r_data_out);
-		else
+			ceph_osdc_msg_data_set(req->r_request,
+						src->extent.osd_data);
+		} else {
  			WARN_ON(src->extent.osd_data != &req->r_data_in);
+			ceph_osdc_msg_data_set(req->r_reply,
+						src->extent.osd_data);
+		}
  		break;
  	case CEPH_OSD_OP_CALL:
  		dst->cls.class_len = src->cls.class_len;
  		dst->cls.method_len = src->cls.method_len;
  		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
  		WARN_ON(src->cls.response_data != &req->r_data_in);
+		ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
  		WARN_ON(src->cls.request_info != &req->r_data_out);
+		ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
  		BUG_ON(src->cls.request_info->type !=
  					CEPH_OSD_DATA_TYPE_PAGELIST);
  		request_data_len = src->cls.request_info->pagelist->length;
net/ceph/osd_client.c +0 -27
@@ -1930,28 +1956,6 @@ bad:
  	return;
  }
 
-static void ceph_osdc_msg_data_set(struct ceph_msg *msg,
-				struct ceph_osd_data *osd_data)
-{
-	u64 length = ceph_osd_data_length(osd_data);
-
-	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES) {
-		BUG_ON(length > (u64) SIZE_MAX);
-		if (length)
-			ceph_msg_data_set_pages(msg, osd_data->pages,
-					length, osd_data->alignment);
-	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGELIST) {
-		BUG_ON(!length);
-		ceph_msg_data_set_pagelist(msg, osd_data->pagelist);
-#ifdef CONFIG_BLOCK
-	} else if (osd_data->type == CEPH_OSD_DATA_TYPE_BIO) {
-		ceph_msg_data_set_bio(msg, osd_data->bio, length);
-#endif
-	} else {
-		BUG_ON(osd_data->type != CEPH_OSD_DATA_TYPE_NONE);
-	}
-}
-
  /*
   * build new request AND message
   *
@@ -1967,11 +1971,6 @@ void ceph_osdc_build_request(struct
 ceph_osd_request *req, u64 off,
  	u64 data_len;
  	unsigned int i;
 
-	/* Set up response incoming data and request outgoing data fields */
-
-	ceph_osdc_msg_data_set(req->r_reply, &req->r_data_in);
-	ceph_osdc_msg_data_set(req->r_request, &req->r_data_out);
-
  	req->r_snapid = snap_id;
  	req->r_snapc = ceph_get_snap_context(snapc);
author
#39 lore raw

PATCH 20/20 libceph: kill off osd request r_data_in and r_data_out

In reply to Alex Elder

Finally! Convert the osd op data pointers into real structures, and make the switch over to using them instead of having all ops share the in and/or out data structures in the osd request.

Set up a new function to traverse the set of ops and release any data associated with them (pages).

This and the patches leading up to it resolve:

    http://tracker.ceph.com/issues/4657
---
 include/linux/ceph/osd_client.h |    9 ++---
 net/ceph/osd_client.c           |   79
+++++++++++++++++++--------------------
 2 files changed, 42 insertions(+), 46 deletions(-)
(quoted diff) +0 -0

b/include/linux/ceph/osd_client.h
index dd4ca4ba..4ec46c0 100644

include/linux/ceph/osd_client.h +3 -6
@@ -87,14 +87,14 @@ struct ceph_osd_req_op {
 			u64 offset, length;
 			u64 truncate_size;
 			u32 truncate_seq;
-			struct ceph_osd_data *osd_data;
+			struct ceph_osd_data osd_data;
 		} extent;
 		struct {
 			const char *class_name;
 			const char *method_name;
 			const void *request_data;
-			struct ceph_osd_data *request_info;
-			struct ceph_osd_data *response_data;
+			struct ceph_osd_data request_info;
+			struct ceph_osd_data response_data;
 			u32 request_data_len;
 			__u8 class_len;
 			__u8 method_len;
@@ -164,9 +164,6 @@ struct ceph_osd_request {
 
 	struct ceph_file_layout r_file_layout;
 	struct ceph_snap_context *r_snapc;    /* snap context for writes */
-
-	struct ceph_osd_data r_data_in;
-	struct ceph_osd_data r_data_out;
 };
 
 struct ceph_osd_event {
net/ceph/osd_client.c +0 -0
index cc4003f..2562e4e 100644
@@ -121,8 +121,7 @@ osd_req_op_extent_osd_data(struct ceph_osd_request

*osd_req,
{
BUG_ON(which >= osd_req->r_num_ops);

- /* return &osd_req->r_ops[which].extent.osd_data; */
- return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
+ return &osd_req->r_ops[which].extent.osd_data;
}
EXPORT_SYMBOL(osd_req_op_extent_osd_data);

@@ -132,8 +131,7 @@ osd_req_op_cls_request_info(struct ceph_osd_request *osd_req,
{
BUG_ON(which >= osd_req->r_num_ops);

- /* return &osd_req->r_ops[which].cls.request_info; */
- return &osd_req->r_data_out; /* Request data is outgoing */
+ return &osd_req->r_ops[which].cls.request_info;
}
EXPORT_SYMBOL(osd_req_op_cls_request_info); /* ??? */

@@ -143,8 +141,7 @@ osd_req_op_cls_response_data(struct ceph_osd_request *osd_req,
{
BUG_ON(which >= osd_req->r_num_ops);

- /* return &osd_req->r_ops[which].cls.response_data; */
- return &osd_req->r_data_in; /* Response data is incoming */
+ return &osd_req->r_ops[which].cls.response_data;
}
EXPORT_SYMBOL(osd_req_op_cls_response_data); /* ??? */

@@ -158,9 +155,6 @@ void osd_req_op_extent_osd_data_pages(struct ceph_osd_request *osd_req,
osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request); ceph_osd_data_pages_init(osd_data, pages, length, alignment,
pages_from_pool, own_pages);
-
- osd_req->r_ops[which].extent.osd_data =
- osd_req_op_extent_osd_data(osd_req, which, write_request);
}
EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);

@@ -172,9 +166,6 @@ void osd_req_op_extent_osd_data_pagelist(struct ceph_osd_request *osd_req,

osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request); ceph_osd_data_pagelist_init(osd_data, pagelist);
-
- osd_req->r_ops[which].extent.osd_data =
- osd_req_op_extent_osd_data(osd_req, which, write_request);
}
EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);

@@ -187,9 +178,6 @@ void osd_req_op_extent_osd_data_bio(struct ceph_osd_request *osd_req,

osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request); ceph_osd_data_bio_init(osd_data, bio, bio_length);
-
- osd_req->r_ops[which].extent.osd_data =
- osd_req_op_extent_osd_data(osd_req, which, write_request);
}
EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
#endif /* CONFIG_BLOCK */
@@ -202,9 +190,6 @@ static void osd_req_op_cls_request_info_pagelist(

osd_data = osd_req_op_cls_request_info(osd_req, which); ceph_osd_data_pagelist_init(osd_data, pagelist);
-
- osd_req->r_ops[which].cls.request_info =
- osd_req_op_cls_request_info(osd_req, which);
}

void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
@@ -216,9 +201,6 @@ void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
osd_data = osd_req_op_cls_response_data(osd_req, which); ceph_osd_data_pages_init(osd_data, pages, length, alignment,
pages_from_pool, own_pages);
-
- osd_req->r_ops[which].cls.response_data =
- osd_req_op_cls_response_data(osd_req, which);
}
EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);

@@ -241,18 +223,39 @@ static u64 ceph_osd_data_length(struct ceph_osd_data *osd_data)
}
}

+
static void ceph_osd_data_release(struct ceph_osd_data *osd_data) {
- if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
- return;
-
- if (osd_data->own_pages) {
+ if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES && osd_data->own_pages) {
int num_pages;

num_pages = calc_pages_for((u64)osd_data->alignment,
(u64)osd_data->length);
ceph_release_page_vector(osd_data->pages, num_pages);
}
+ ceph_osd_data_init(osd_data);
+}
+
+static void osd_req_op_data_release(struct ceph_osd_request *osd_req,
+ unsigned int which)
+{
+ struct ceph_osd_req_op *op;
+
+ BUG_ON(which >= osd_req->r_num_ops);
+ op = &osd_req->r_ops[which];
+
+ switch (op->op) {
+ case CEPH_OSD_OP_READ:
+ case CEPH_OSD_OP_WRITE:
+ ceph_osd_data_release(&op->extent.osd_data);
+ break;
+ case CEPH_OSD_OP_CALL:
+ ceph_osd_data_release(&op->cls.request_info);
+ ceph_osd_data_release(&op->cls.response_data);
+ break;
+ default:
+ break;
+ }
}

/*
@@ -261,6 +264,7 @@ static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
void ceph_osdc_release_request(struct kref *kref) {
struct ceph_osd_request *req;
+ unsigned int which;

req = container_of(kref, struct ceph_osd_request, r_kref); if (req->r_request)
@@ -270,8 +274,8 @@ void ceph_osdc_release_request(struct kref *kref)
ceph_msg_put(req->r_reply);
}

- ceph_osd_data_release(&req->r_data_in);
- ceph_osd_data_release(&req->r_data_out);
+ for (which = 0; which < req->r_num_ops; which++)
+ osd_req_op_data_release(req, which);

ceph_put_snap_context(req->r_snapc);
if (req->r_mempool)
@@ -595,27 +599,22 @@ static u64 osd_req_encode_op(struct ceph_osd_request *req,
cpu_to_le64(src->extent.truncate_size);
dst->extent.truncate_seq =
cpu_to_le32(src->extent.truncate_seq);
- if (src->op == CEPH_OSD_OP_WRITE) {
- WARN_ON(src->extent.osd_data != &req->r_data_out);
+ if (src->op == CEPH_OSD_OP_WRITE)
ceph_osdc_msg_data_set(req->r_request,
- src->extent.osd_data);
- } else {
- WARN_ON(src->extent.osd_data != &req->r_data_in);
+ &src->extent.osd_data);
+ else
ceph_osdc_msg_data_set(req->r_reply,
- src->extent.osd_data);
- }
+ &src->extent.osd_data);
break;
case CEPH_OSD_OP_CALL:
dst->cls.class_len = src->cls.class_len; dst->cls.method_len = src->cls.method_len; dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
- WARN_ON(src->cls.response_data != &req->r_data_in);
- ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
- WARN_ON(src->cls.request_info != &req->r_data_out);
- ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
- BUG_ON(src->cls.request_info->type !=
+ ceph_osdc_msg_data_set(req->r_reply, &src->cls.response_data);
+ ceph_osdc_msg_data_set(req->r_request, &src->cls.request_info);
+ BUG_ON(src->cls.request_info.type !=
CEPH_OSD_DATA_TYPE_PAGELIST);
- request_data_len = src->cls.request_info->pagelist->length;
+ request_data_len = src->cls.request_info.pagelist->length;
break;
case CEPH_OSD_OP_STARTSYNC:
break;

Signature · 1 line
1.7.9.5
Reviewed-by on v1 20/20
#40 lore raw

Re: [PATCH 20/20] libceph: kill off osd request r_data_in and r_data_out

Quoted 244 lines · Alex Elder

Finally! Convert the osd op data pointers into real structures, and make the switch over to using them instead of having all ops share the in and/or out data structures in the osd request.

Set up a new function to traverse the set of ops and release any data associated with them (pages).

This and the patches leading up to it resolve:

     http://tracker.ceph.com/issues/4657
---
  include/linux/ceph/osd_client.h |    9 ++---
  net/ceph/osd_client.c           |   79
+++++++++++++++++++--------------------
  2 files changed, 42 insertions(+), 46 deletions(-)
include/linux/ceph/osd_client.h +3 -6
index dd4ca4ba..4ec46c0 100644
@@ -87,14 +87,14 @@ struct ceph_osd_req_op {
  			u64 offset, length;
  			u64 truncate_size;
  			u32 truncate_seq;
-			struct ceph_osd_data *osd_data;
+			struct ceph_osd_data osd_data;
  		} extent;
  		struct {
  			const char *class_name;
  			const char *method_name;
  			const void *request_data;
-			struct ceph_osd_data *request_info;
-			struct ceph_osd_data *response_data;
+			struct ceph_osd_data request_info;
+			struct ceph_osd_data response_data;
  			u32 request_data_len;
  			__u8 class_len;
  			__u8 method_len;
@@ -164,9 +164,6 @@ struct ceph_osd_request {
 
  	struct ceph_file_layout r_file_layout;
  	struct ceph_snap_context *r_snapc;    /* snap context for writes */
-
-	struct ceph_osd_data r_data_in;
-	struct ceph_osd_data r_data_out;
  };
 
  struct ceph_osd_event {
net/ceph/osd_client.c +3 -15
index cc4003f..2562e4e 100644
@@ -121,8 +121,7 @@ osd_req_op_extent_osd_data(struct ceph_osd_request
 *osd_req,
  {
  	BUG_ON(which >= osd_req->r_num_ops);
 
-	/* return &osd_req->r_ops[which].extent.osd_data; */
-	return write_request ? &osd_req->r_data_out : &osd_req->r_data_in;
+	return &osd_req->r_ops[which].extent.osd_data;
  }
  EXPORT_SYMBOL(osd_req_op_extent_osd_data);
@@ -132,8 +131,7 @@ osd_req_op_cls_request_info(struct ceph_osd_request
 *osd_req,
  {
  	BUG_ON(which >= osd_req->r_num_ops);
 
-	/* return &osd_req->r_ops[which].cls.request_info; */
-	return &osd_req->r_data_out;	/* Request data is outgoing */
+	return &osd_req->r_ops[which].cls.request_info;
  }
  EXPORT_SYMBOL(osd_req_op_cls_request_info);	/* ??? */
@@ -143,8 +141,7 @@ osd_req_op_cls_response_data(struct ceph_osd_request
 *osd_req,
  {
  	BUG_ON(which >= osd_req->r_num_ops);
 
-	/* return &osd_req->r_ops[which].cls.response_data; */
-	return &osd_req->r_data_in;	/* Response data is incoming */
+	return &osd_req->r_ops[which].cls.response_data;
  }
  EXPORT_SYMBOL(osd_req_op_cls_response_data);	/* ??? */
@@ -158,9 +155,6 @@ void osd_req_op_extent_osd_data_pages(struct
 ceph_osd_request *osd_req,
  	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
  	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
  				pages_from_pool, own_pages);
-
-	osd_req->r_ops[which].extent.osd_data =
-		osd_req_op_extent_osd_data(osd_req, which, write_request);
  }
  EXPORT_SYMBOL(osd_req_op_extent_osd_data_pages);
@@ -172,9 +166,6 @@ void osd_req_op_extent_osd_data_pagelist(struct
 ceph_osd_request *osd_req,
 
  	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
  	ceph_osd_data_pagelist_init(osd_data, pagelist);
-
-	osd_req->r_ops[which].extent.osd_data =
-		osd_req_op_extent_osd_data(osd_req, which, write_request);
  }
  EXPORT_SYMBOL(osd_req_op_extent_osd_data_pagelist);
@@ -187,9 +178,6 @@ void osd_req_op_extent_osd_data_bio(struct
 ceph_osd_request *osd_req,
 
  	osd_data = osd_req_op_extent_osd_data(osd_req, which, write_request);
  	ceph_osd_data_bio_init(osd_data, bio, bio_length);
-
-	osd_req->r_ops[which].extent.osd_data =
-		osd_req_op_extent_osd_data(osd_req, which, write_request);
  }
  EXPORT_SYMBOL(osd_req_op_extent_osd_data_bio);
  #endif /* CONFIG_BLOCK */
net/ceph/osd_client.c +25 -10
@@ -202,9 +190,6 @@ static void osd_req_op_cls_request_info_pagelist(
 
  	osd_data = osd_req_op_cls_request_info(osd_req, which);
  	ceph_osd_data_pagelist_init(osd_data, pagelist);
-
-	osd_req->r_ops[which].cls.request_info =
-		osd_req_op_cls_request_info(osd_req, which);
  }
 
  void osd_req_op_cls_response_data_pages(struct ceph_osd_request *osd_req,
@@ -216,9 +201,6 @@ void osd_req_op_cls_response_data_pages(struct
 ceph_osd_request *osd_req,
  	osd_data = osd_req_op_cls_response_data(osd_req, which);
  	ceph_osd_data_pages_init(osd_data, pages, length, alignment,
  				pages_from_pool, own_pages);
-
-	osd_req->r_ops[which].cls.response_data =
-		osd_req_op_cls_response_data(osd_req, which);
  }
  EXPORT_SYMBOL(osd_req_op_cls_response_data_pages);
@@ -241,18 +223,39 @@ static u64 ceph_osd_data_length(struct
 ceph_osd_data *osd_data)
  	}
  }
 
+
  static void ceph_osd_data_release(struct ceph_osd_data *osd_data)
  {
-	if (osd_data->type != CEPH_OSD_DATA_TYPE_PAGES)
-		return;
-
-	if (osd_data->own_pages) {
+	if (osd_data->type == CEPH_OSD_DATA_TYPE_PAGES && osd_data->own_pages) {
  		int num_pages;
 
  		num_pages = calc_pages_for((u64)osd_data->alignment,
  						(u64)osd_data->length);
  		ceph_release_page_vector(osd_data->pages, num_pages);
  	}
+	ceph_osd_data_init(osd_data);
+}
+
+static void osd_req_op_data_release(struct ceph_osd_request *osd_req,
+			unsigned int which)
+{
+	struct ceph_osd_req_op *op;
+
+	BUG_ON(which >= osd_req->r_num_ops);
+	op = &osd_req->r_ops[which];
+
+	switch (op->op) {
+	case CEPH_OSD_OP_READ:
+	case CEPH_OSD_OP_WRITE:
+		ceph_osd_data_release(&op->extent.osd_data);
+		break;
+	case CEPH_OSD_OP_CALL:
+		ceph_osd_data_release(&op->cls.request_info);
+		ceph_osd_data_release(&op->cls.response_data);
+		break;
+	default:
+		break;
+	}
  }
 

/*

net/ceph/osd_client.c +1 -0
@@ -261,6 +264,7 @@ static void ceph_osd_data_release(struct
 ceph_osd_data *osd_data)
  void ceph_osdc_release_request(struct kref *kref)
  {
  	struct ceph_osd_request *req;
+	unsigned int which;
 
  	req = container_of(kref, struct ceph_osd_request, r_kref);

if (req->r_request)

net/ceph/osd_client.c +10 -15
@@ -270,8 +274,8 @@ void ceph_osdc_release_request(struct kref *kref)
  		ceph_msg_put(req->r_reply);
  	}
 
-	ceph_osd_data_release(&req->r_data_in);
-	ceph_osd_data_release(&req->r_data_out);
+	for (which = 0; which < req->r_num_ops; which++)
+		osd_req_op_data_release(req, which);
 
  	ceph_put_snap_context(req->r_snapc);
  	if (req->r_mempool)
@@ -595,27 +599,22 @@ static u64 osd_req_encode_op(struct
 ceph_osd_request *req,
  			cpu_to_le64(src->extent.truncate_size);
  		dst->extent.truncate_seq =
  			cpu_to_le32(src->extent.truncate_seq);
-		if (src->op == CEPH_OSD_OP_WRITE) {
-			WARN_ON(src->extent.osd_data != &req->r_data_out);
+		if (src->op == CEPH_OSD_OP_WRITE)
  			ceph_osdc_msg_data_set(req->r_request,
-						src->extent.osd_data);
-		} else {
-			WARN_ON(src->extent.osd_data != &req->r_data_in);
+						&src->extent.osd_data);
+		else
  			ceph_osdc_msg_data_set(req->r_reply,
-						src->extent.osd_data);
-		}
+						&src->extent.osd_data);
  		break;
  	case CEPH_OSD_OP_CALL:
  		dst->cls.class_len = src->cls.class_len;
  		dst->cls.method_len = src->cls.method_len;
  		dst->cls.indata_len = cpu_to_le32(src->cls.request_data_len);
-		WARN_ON(src->cls.response_data != &req->r_data_in);
-		ceph_osdc_msg_data_set(req->r_reply, src->cls.response_data);
-		WARN_ON(src->cls.request_info != &req->r_data_out);
-		ceph_osdc_msg_data_set(req->r_request, src->cls.request_info);
-		BUG_ON(src->cls.request_info->type !=
+		ceph_osdc_msg_data_set(req->r_reply, &src->cls.response_data);
+		ceph_osdc_msg_data_set(req->r_request, &src->cls.request_info);
+		BUG_ON(src->cls.request_info.type !=
  					CEPH_OSD_DATA_TYPE_PAGELIST);
-		request_data_len = src->cls.request_info->pagelist->length;
+		request_data_len = src->cls.request_info.pagelist->length;
  		break;
  	case CEPH_OSD_OP_STARTSYNC:

break;