Skip to contentSkip to conversation

handling Rerror without copy_from_iter_full()

RFC In review Al Viro 2 participants 4 messages Jun 8, 2022 – Jun 11, 2022

Subsystems (2)
lore
Share & tools
lore

Al Viro

author maintainer
#0 lore raw

RFC 1/1 handling Rerror without copy_from_iter_full()

As it is, p9_client_zc_rpc()/p9_check_zc_errors() is playing fast
and loose with copy_from_iter_full().

Background: reading from file is done by sending Tread request.
Response consists of fixed-sized header (including the amount of data actually read) followed by the data itself.

For zero-copy case we arrange the things so that the first 11
bytes of reply go into the fixed-sized buffer, with the rest going straight into the pages we want to read into.

What makes the things inconvenient is that sglist describing
what should go where has to be set *before* the reply arrives. As the result, if reply is an error, the things get interesting. Success is

	size[4] Rread tag[2] count[4] data[count]
For error layout varies depending upon the protocol variant -
in original 9P and 9P2000 it's
	size[4] Rerror tag[2] len[2] error[len]
in 9P2000.U
	size[4] Rerror tag[2] len[2] error[len] errno[4]
in 9P2000.L
	size[4] Rlerror tag[2] errno[4]

The last case is nice and simple - we have an 11-byte response that fits into the fixed-sized buffer we hoped to get an Rread into. In other two, though, we get a variable-length string spill into the pages we'd prepared for the data to be read.

Had that been in fixed-sized buffer (which is actually 4K), we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages.

The trouble is, by the time we get around to that, the references to pages in question are already dropped. As the result, p9_zc_check_errors() tries to get the data using copy_from_iter_full(). Unfortunately, the iov_iter it's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's an ITER_PIPE one, copy_from_iter_full() will simply fail.

The thing is, in ->zc_request() itself we have those pages. There it would be a simple matter of memcpy_from_page() into the fixed-sized buffer and it isn't hard to recognize the (rare) case when such copying is needed. That way we get rid of p9_zc_check_errors() entirely
- p9_check_errors() can be used instead both for zero-copy and non-zero-copy cases.

Do you see any problems with the variant below?

---

net/9p/client.c +1 -85
index 8bba0d9cf975..d403085b9ef5 100644
@@ -550,90 +550,6 @@ static int p9_check_errors(struct p9_client *c, struct p9_req_t *req)
 	return err;
 }
 
-/**
- * p9_check_zc_errors - check 9p packet for error return and process it
- * @c: current client instance
- * @req: request to parse and check for error conditions
- * @uidata: external buffer containing error
- * @in_hdrlen: Size of response protocol buffer.
- *
- * returns error code if one is discovered, otherwise returns 0
- *
- * this will have to be more complicated if we have multiple
- * error packet types
- */
-
-static int p9_check_zc_errors(struct p9_client *c, struct p9_req_t *req,
-			      struct iov_iter *uidata, int in_hdrlen)
-{
-	int err;
-	int ecode;
-	s8 type;
-	char *ename = NULL;
-
-	err = p9_parse_header(&req->rc, NULL, &type, NULL, 0);
-	/* dump the response from server
-	 * This should be after parse_header which poplulate pdu_fcall.
-	 */
-	trace_9p_protocol_dump(c, &req->rc);
-	if (err) {
-		p9_debug(P9_DEBUG_ERROR, "couldn't parse header %d\n", err);
-		return err;
-	}
-
-	if (type != P9_RERROR && type != P9_RLERROR)
-		return 0;
-
-	if (!p9_is_proto_dotl(c)) {
-		/* Error is reported in string format */
-		int len;
-		/* 7 = header size for RERROR; */
-		int inline_len = in_hdrlen - 7;
-
-		len = req->rc.size - req->rc.offset;
-		if (len > (P9_ZC_HDR_SZ - 7)) {
-			err = -EFAULT;
-			goto out_err;
-		}
-
-		ename = &req->rc.sdata[req->rc.offset];
-		if (len > inline_len) {
-			/* We have error in external buffer */
-			if (!copy_from_iter_full(ename + inline_len,
-						 len - inline_len, uidata)) {
-				err = -EFAULT;
-				goto out_err;
-			}
-		}
-		ename = NULL;
-		err = p9pdu_readf(&req->rc, c->proto_version, "s?d",
-				  &ename, &ecode);
-		if (err)
-			goto out_err;
-
-		if (p9_is_proto_dotu(c) && ecode < 512)
-			err = -ecode;
-
-		if (!err) {
-			err = p9_errstr2errno(ename, strlen(ename));
-
-			p9_debug(P9_DEBUG_9P, "<<< RERROR (%d) %s\n",
-				 -ecode, ename);
-		}
-		kfree(ename);
-	} else {
-		err = p9pdu_readf(&req->rc, c->proto_version, "d", &ecode);
-		err = -ecode;
-
-		p9_debug(P9_DEBUG_9P, "<<< RLERROR (%d)\n", -ecode);
-	}
-	return err;
-
-out_err:
-	p9_debug(P9_DEBUG_ERROR, "couldn't parse error%d\n", err);
-	return err;
-}
-
 static struct p9_req_t *
 p9_client_rpc(struct p9_client *c, int8_t type, const char *fmt, ...);
 
@@ -874,7 +790,7 @@ static struct p9_req_t *p9_client_zc_rpc(struct p9_client *c, int8_t type,
 	if (err < 0)
 		goto reterr;
 
-	err = p9_check_zc_errors(c, req, uidata, in_hdrlen);
+	err = p9_check_errors(c, req);
 	trace_9p_client_res(c, type, req->rc.tag, err);
 	if (!err)
 		return req;
net/9p/trans_virtio.c +34 -0
index b24a4fb0f0a2..2a210c2f8e40 100644
@@ -377,6 +377,35 @@ static int p9_get_mapped_pages(struct virtio_chan *chan,
 	}
 }
 
+static void handle_rerror(struct p9_req_t *req, int in_hdr_len,
+			  size_t offs, struct page **pages)
+{
+	unsigned size, n;
+	void *to = req->rc.sdata + in_hdr_len;
+
+	// Fits entirely into the static data?  Nothing to do.
+	if (req->rc.size < in_hdr_len)
+		return;
+
+	// Really long error message?  Tough, truncate the reply.  Might get
+	// rejected (we can't be arsed to adjust the size encoded in header,
+	// or string size for that matter), but it wouldn't be anything valid
+	// anyway.
+	if (unlikely(req->rc.size > P9_ZC_HDR_SZ))
+		req->rc.size = P9_ZC_HDR_SZ;
+
+	// data won't span more than two pages
+	size = req->rc.size - in_hdr_len;
+	n = PAGE_SIZE - offs;
+	if (size > n) {
+		memcpy_from_page(to, *pages++, offs, n);
+		offs = 0;
+		to += n;
+		size -= n;
+	}
+	memcpy_from_page(to, *pages, offs, size);
+}
+
 /**
  * p9_virtio_zc_request - issue a zero copy request
  * @client: client instance issuing the request
@@ -503,6 +532,11 @@ p9_virtio_zc_request(struct p9_client *client, struct p9_req_t *req,
 	kicked = 1;
 	p9_debug(P9_DEBUG_TRANS, "virtio request kicked\n");
 	err = wait_event_killable(req->wq, req->status >= REQ_STATUS_RCVD);
+	// RERROR needs reply (== error string) in static data
+	if (req->status == REQ_STATUS_RCVD &&
+	    unlikely(req->rc.sdata[4] == P9_RERROR))
+		handle_rerror(req, in_hdr_len, offs, in_pages);
+
 	/*
 	 * Non kernel buffers are pinned, unpin them
 	 */
#1 lore raw

Al Viro wrote on Wed, Jun 08, 2022 at 05:41:46PM +0000:

Quoted 10 lines

As it is, p9_client_zc_rpc()/p9_check_zc_errors() is playing fast
and loose with copy_from_iter_full().

Background: reading from file is done by sending Tread request.
Response consists of fixed-sized header (including the amount of data actually read) followed by the data itself.

For zero-copy case we arrange the things so that the first 11
bytes of reply go into the fixed-sized buffer, with the rest going straight into the pages we want to read into.

Ugh... zc really needs something like direct data placement NFS/RDMA has been doing... (But that's just not possible without extending the protocol)

Quoted 13 lines

What makes the things inconvenient is that sglist describing
what should go where has to be set *before* the reply arrives. As the result, if reply is an error, the things get interesting. Success is

	size[4] Rread tag[2] count[4] data[count]
For error layout varies depending upon the protocol variant -
in original 9P and 9P2000 it's
	size[4] Rerror tag[2] len[2] error[len]
in 9P2000.U
	size[4] Rerror tag[2] len[2] error[len] errno[4]
in 9P2000.L
	size[4] Rlerror tag[2] errno[4]
Quoted 4 lines

The last case is nice and simple - we have an 11-byte response that fits into the fixed-sized buffer we hoped to get an Rread into. In other two, though, we get a variable-length string spill into the pages we'd prepared for the data to be read.

That makes me wonder just how much use we get for the legacy protocols -- I guess we do have some but all the filesystem-y implementations that I would expect to be main users for large IOs/zc are 9P2000.L as far as I know -- especially considering virtio is pretty much limited to qemu? Are there other 9p virtio servers?

So would it make sense to just say "not .L? tough luck, no zc", or am I just being lazy?

Quoted 11 lines

Had that been in fixed-sized buffer (which is actually 4K), we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages.

The trouble is, by the time we get around to that, the references to pages in question are already dropped. As the result, p9_zc_check_errors() tries to get the data using copy_from_iter_full(). Unfortunately, the iov_iter it's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's an ITER_PIPE one, copy_from_iter_full() will simply fail.

Silly question, in case of a pipe we'll have written something we shouldn't have, or is it not gone yet until we actually finish the IO with iov_iter_advance?
(my understanding is that reader won't have anything to read until someone else does the advance, and that someone else will have overwritten the data with their own content so no garbage wlll be read)

Quoted 8 lines

The thing is, in ->zc_request() itself we have those pages. There it would be a simple matter of memcpy_from_page() into the fixed-sized buffer and it isn't hard to recognize the (rare) case when such copying is needed. That way we get rid of p9_zc_check_errors() entirely
- p9_check_errors() can be used instead both for zero-copy and non-zero-copy cases.

Do you see any problems with the variant below?

... With that said though your approach here definitely looks better than what we currently have -- my main issue is that truncating is fine for the original 9p2000 but for .U you'd be losing the ecode that we currently trust for being errno-compatible.
It's *probably* not a problem in practice, but preserving that errno would theorically make us look for the page where the last few bytes went to and overwrite the end of the string with it but that's starting to be ugly.

Anyway even not doing that is probably better than reading from something we no longer own; but I'm still thinking just refusing non-.L variants to zc calls is a better decision long term.

Signature · 1 line
Dominique
author maintainer
#2 lore raw
Quoted 6 lines · Dominique Martinet

That makes me wonder just how much use we get for the legacy protocols -- I guess we do have some but all the filesystem-y implementations that I would expect to be main users for large IOs/zc are 9P2000.L as far as I know -- especially considering virtio is pretty much limited to qemu? Are there other 9p virtio servers?

Client can trivially force its use - -o version=9p2000.u and there you go...

Quoted 15 lines

So would it make sense to just say "not .L? tough luck, no zc", or am I just being lazy?

Quoted 11 lines

Had that been in fixed-sized buffer (which is actually 4K), we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages.

The trouble is, by the time we get around to that, the references to pages in question are already dropped. As the result, p9_zc_check_errors() tries to get the data using copy_from_iter_full(). Unfortunately, the iov_iter it's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's an ITER_PIPE one, copy_from_iter_full() will simply fail.

Quoted 6 lines

Silly question, in case of a pipe we'll have written something we shouldn't have, or is it not gone yet until we actually finish the IO with iov_iter_advance?
(my understanding is that reader won't have anything to read until someone else does the advance, and that someone else will have overwritten the data with their own content so no garbage wlll be read)

More than that, pipe is locked through ->read_iter(), so transient garbage in it doesn't matter - we will either advance it by zero (it's an error) or, with iov_iter_get_pages_alloc() switching to advancing variant, we'll revert by the amount it had reserved there (error is the same as extremely short read).

Quoted 4 lines

... With that said though your approach here definitely looks better than what we currently have -- my main issue is that truncating is fine for the original 9p2000 but for .U you'd be losing the ecode that we currently trust for being errno-compatible.

No, we wouldn't - this

                len = req->rc.size - req->rc.offset;
		if (len > (P9_ZC_HDR_SZ - 7)) {
			err = -EFAULT;
			goto out_err;
		}
in p9_check_zc_errors() means that mainline won't even look at that value.
And we'll get the same -EFAULT when we get to
                err = p9pdu_readf(&req->rc, c->proto_version, "s?d",
				  &ename, &ecode);

in p9_check_errors() and see that the length of string is too large to fit into (truncated) reply.

#3 lore raw

Al Viro wrote on Sat, Jun 11, 2022 at 03:19:07PM +0000:

Quoted 9 lines
Quoted 6 lines

That makes me wonder just how much use we get for the legacy protocols -- I guess we do have some but all the filesystem-y implementations that I would expect to be main users for large IOs/zc are 9P2000.L as far as I know -- especially considering virtio is pretty much limited to qemu? Are there other 9p virtio servers?

Client can trivially force its use - -o version=9p2000.u and there you go...

Well, yes, but qemu supports .L -- even if we arbitrarily decide .u won't go through the zero-copy path it's not going to harm the VM case.

If zc had been a thing for e.g. trans_fd there are plenty of servers that could still be only used with older variants, but I don't think we have to try supporting 9p2000.u + zc if it's a burden in the code.

Quoted 7 lines
Quoted 6 lines

Silly question, in case of a pipe we'll have written something we shouldn't have, or is it not gone yet until we actually finish the IO with iov_iter_advance?
(my understanding is that reader won't have anything to read until someone else does the advance, and that someone else will have overwritten the data with their own content so no garbage wlll be read)

Quoted 5 lines

More than that, pipe is locked through ->read_iter(), so transient garbage in it doesn't matter - we will either advance it by zero (it's an error) or, with iov_iter_get_pages_alloc() switching to advancing variant, we'll revert by the amount it had reserved there (error is the same as extremely short read).

Ok, thanks

Quoted 15 lines
Quoted 4 lines

... With that said though your approach here definitely looks better than what we currently have -- my main issue is that truncating is fine for the original 9p2000 but for .U you'd be losing the ecode that we currently trust for being errno-compatible.

No, we wouldn't - this

                len = req->rc.size - req->rc.offset;
		if (len > (P9_ZC_HDR_SZ - 7)) {
			err = -EFAULT;
			goto out_err;
		}
in p9_check_zc_errors() means that mainline won't even look at that value.
And we'll get the same -EFAULT when we get to
                err = p9pdu_readf(&req->rc, c->proto_version, "s?d",
				  &ename, &ecode);
Quoted 2 lines

in p9_check_errors() and see that the length of string is too large to fit into (truncated) reply.

Right, I forgot the string itself has a size, so we're not looking at the last bytes but something that is no longer there and readf will fail. Ok.

Sure, I don't see any other problem with this code.
I still think it's complexity we don't really need, but it's better than what we have -- care to resend it as a real patch? and I'll apply/run through some basic tests with qemu+2000u a bit.
(well, I don't have to wait to run the tests)

Signature · 1 line
Dominique