Skip to contentSkip to conversation

RFC: [PATCH] staging/lustre/llite: fix O_TMPFILE/O_LOV_DELAY_CREATE conflict

DISCUSSION Dilger, Andreas 6 participants 15 messages Feb 10, 2014 – Feb 11, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw
Quoted 13 lines · Andreas
Quoted 12 lines · Andreas Dilger

In kernel 3.11 O_TMPFILE was introduced, but the open flag value conflicts with the O_LOV_DELAY_CREATE flag 020000000 previously used by Lustre-aware applications. O_LOV_DELAY_CREATE allows applications to defer file layout and object creation from open time (the default) until it can instead be specified by the application using an ioctl.

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag or define a Lustre-specific flag that isn't of use to most/any other filesystems, use (O_NOCTTY|FASYNC) as the new value. These flag are not meaningful for newly-created regular files and should be OK since O_LOV_DELAY_CREATE is only meaningful for new files.

Quoted 5 lines
Quoted 5 lines

I looked into using O_ACCMODE/FMODE_WRITE_IOCTL, which allows calling ioctl() on the minimally-opened fd and is close to what is needed, but that doesn't allow specifying the actual read or write mode for the file, and fcntl(F_SETFL) doesn't allow O_RDONLY/O_WRONLY/O_RDWR to be set after the file is opened.

Al, Christoph,
any comments on this approach?

Quoted 155 lines

A few extra comments here that I wasn't sure should be in the commit comment.

The main goal of the O_LOV_DELAY_CREATE flag is to allow the file to be opened in a "preliminary" manner to allow the application to specify the layout of the file across the Lustre storage targets (e.g. whether the app has millions of separate files, each one written to a single server, or there is a single huge file spread across all of the servers, or some combination of the two, if it is RAID-0 or RAID-1, or whatever).

I'm open to a separate flag value for this, but I don't know if that is of much interest to other filesystems since they don't have similar needs. We want to avoid the need to have lots of syscalls to do this, since they translate into multiple RPCs that we want to avoid when creating potentially
millions of files over the network.

Cheers, Andreas

Quoted 136 lines

Lustre-change: http://review.whamcloud.com/8312
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-4209

---

.../lustre/lustre/include/lustre/lustre_user.h     |   12 ++++------
drivers/staging/lustre/lustre/include/lustre_mdc.h |   11 ++++++++++
drivers/staging/lustre/lustre/llite/file.c         |   21
++++++++++---------
drivers/staging/lustre/lustre/mdc/mdc_lib.c        |    2 +-
4 files changed, 28 insertions(+), 18 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_user.h +0 -1
index 6b6c0240..91f22a8 100644
@@ -265,13 +265,11 @@ struct ost_id {
 
 #define MAX_OBD_NAME 128 /* If this changes, a NEW ioctl must be added
 */
 
-/* Hopefully O_LOV_DELAY_CREATE does not conflict with standard O_xxx

flags.
- * Previously it was defined as 0100000000 and conflicts with FMODE_NONOTIFY
- * which was added since kernel 2.6.36, so we redefine it as 020000000.
- * To be compatible with old version's statically linked binary, finally we
- * define it as (020000000 | 0100000000).
- * */

drivers/staging/lustre/lustre/include/lustre/lustre_user.h +3 -1
-#define O_LOV_DELAY_CREATE      0120000000
+/* Define O_LOV_DELAY_CREATE to be a mask that is not useful for regular
+ * files, but are unlikely to be used in practice and are not harmful if
+ * used incorrectly.  O_NOCTTY and FASYNC are only meaningful for

character
+ * devices and are safe for use on new files (See LU-812, LU-4209). */
+#define O_LOV_DELAY_CREATE (O_NOCTTY | FASYNC)

#define LL_FILE_IGNORE_LOCK 0x00000001
#define LL_FILE_GROUP_LOCKED 0x00000002

drivers/staging/lustre/lustre/include/lustre_mdc.h +11 -0
index c1e0270..468f363 100644
@@ -166,6 +166,17 @@ void it_clear_disposition(struct lookup_intent *it,
 int flag);
 void it_set_disposition(struct lookup_intent *it, int flag);
 int it_open_error(int phase, struct lookup_intent *it);
 
+static inline bool cl_is_lov_delay_create(unsigned int flags)
+{
+	return (flags & O_LOV_DELAY_CREATE) == O_LOV_DELAY_CREATE;
+}
+
+static inline void cl_lov_delay_create_clear(unsigned int *flags)
+{
+	if ((*flags & O_LOV_DELAY_CREATE) == O_LOV_DELAY_CREATE)
+		*flags &= ~O_LOV_DELAY_CREATE;
+}
+
 /** @} mdc */
 

#endif

drivers/staging/lustre/lustre/llite/file.c +6 -7
index c12821a..dc9da77 100644
@@ -671,14 +671,13 @@ restart:
 
 	ll_capa_open(inode);
 
-	if (!lli->lli_has_smd) {
-		if (file->f_flags & O_LOV_DELAY_CREATE ||
-		    !(file->f_mode & FMODE_WRITE)) {
-			CDEBUG(D_INODE, "object creation was delayed\n");
-			GOTO(out_och_free, rc);
-		}
+	if (!lli->lli_has_smd &&
+	    (cl_is_lov_delay_create(file->f_flags) ||
+	     (file->f_mode & FMODE_WRITE) == 0)) {
+		CDEBUG(D_INODE, "object creation was delayed\n");
+		GOTO(out_och_free, rc);
 	}
-	file->f_flags &= ~O_LOV_DELAY_CREATE;
+	cl_lov_delay_create_clear(&file->f_flags);
 	GOTO(out_och_free, rc);
 

out_och_free:

drivers/staging/lustre/lustre/llite/file.c +5 -3
@@ -1381,23 +1380,25 @@ int ll_lov_setstripe_ea_info(struct inode *inode,
 struct file *file,
 		ccc_inode_lsm_put(inode, lsm);
 		CDEBUG(D_IOCTL, "stripe already exists for ino %lu\n",
 		       inode->i_ino);
-		return -EEXIST;
+		GOTO(out, rc = -EEXIST);
 	}
 
 	ll_inode_size_lock(inode);
 	rc = ll_intent_file_open(file, lum, lum_size, &oit);
 	if (rc)
-		GOTO(out, rc);
+		GOTO(out_unlock, rc);
 	rc = oit.d.lustre.it_status;
 	if (rc < 0)
 		GOTO(out_req_free, rc);
 
 	ll_release_openhandle(file->f_dentry, &oit);
 
- out:
+out_unlock:
 	ll_inode_size_unlock(inode);
 	ll_intent_release(&oit);
 	ccc_inode_lsm_put(inode, lsm);
+out:
+	cl_lov_delay_create_clear(&file->f_flags);
 	return rc;

out_req_free:

	ptlrpc_req_finished((struct ptlrpc_request *) oit.d.lustre.it_data);
drivers/staging/lustre/lustre/mdc/mdc_lib.c +1 -1
index 91f6876..5b9f371 100644
@@ -197,7 +197,7 @@ static __u64 mds_pack_open_flags(__u64 flags, __u32
 mode)
 	if (flags & FMODE_EXEC)
 		cr_flags |= MDS_FMODE_EXEC;
 #endif
-	if (flags & O_LOV_DELAY_CREATE)
+	if (cl_is_lov_delay_create(flags))
 		cr_flags |= MDS_OPEN_DELAY_CREATE;
 
Quoted 16 lines
Quoted 5 lines

if (flags & O_NONBLOCK)

Signature · 1 line
1.7.3.4

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
maintainer
#1 lore raw
Quoted 5 lines · Andreas
Quoted 5 lines
Quoted 5 lines

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag or define a Lustre-specific flag that isn't of use to most/any other filesystems, use (O_NOCTTY|FASYNC) as the new value. These flag are not meaningful for newly-created regular files and should be OK since O_LOV_DELAY_CREATE is only meaningful for new files.

*shrug*

I can live with that; it's a kludge, but it's less broken than that explicit constant - that one is a non-starter, since O_... flag values are arch-dependent.

I have another question about what you are doing there - the games
you are playing with crw_pos. Is there any reason not to have ->ki_pos updated immediately in lustre_generic_file_read()/lustre_generic_file_write()?

These two are the only places in the entire tree where
generic_file_aio_{read,write}() does *not* have ppos argument equal to &iocb->ki_pos and I would very much prefer to kill the sucker off.

maintainer
#2 lore raw
Quoted 14 lines · Al Viro
Quoted 5 lines · Andreas
Quoted 5 lines
Quoted 5 lines

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag or define a Lustre-specific flag that isn't of use to most/any other filesystems, use (O_NOCTTY|FASYNC) as the new value. These flag are not meaningful for newly-created regular files and should be OK since O_LOV_DELAY_CREATE is only meaningful for new files.

*shrug*

I can live with that; it's a kludge, but it's less broken than that explicit constant - that one is a non-starter, since O_... flag values are arch-dependent.

Quoted 8 lines

I have another question about what you are doing there - the games
you are playing with crw_pos. Is there any reason not to have ->ki_pos updated immediately in lustre_generic_file_read()/lustre_generic_file_write()?

These two are the only places in the entire tree where
generic_file_aio_{read,write}() does *not* have ppos argument equal to &iocb->ki_pos and I would very much prefer to kill the sucker off.

Ugh... Sorry, I misread that code. Why the devil do you have the pos argument passed to lustre_generic_file_{read,write}() by address, when both proceed to dereference it and pass the value on?

maintainer
#3 lore raw
Quoted 3 lines · Al Viro

Ugh... Sorry, I misread that code. Why the devil do you have the pos argument passed to lustre_generic_file_{read,write}() by address, when both proceed to dereference it and pass the value on?

Egads. Correct me if I'm wrong, please, but it looks like you have

1) ->aio_write() stash its arguments in an off-stack struct and pass it and &iocb->ki_pos to

2) ll_file_io_generic() which calls

3) cl_io_rw_init() which copied ki_pos value passed to it into io->u.ci_rw.crw_pos (in another off-stack struct) and calls

4) cl_io_init() which calls

5) cl_io_init0() which possibly calls a bunch of instances of ->coo_io_init(), which may or may not return 0; I _hope_ that in this case that's what'll happen and we get back to ll_file_io_generic(), where

3') we stash iocb and iov/iovlen into the third off-stack structure (cio) and call

4') cl_io_loop() where we (in a loop) call cl_io_iter_init(), cl_io_lock() and

5') cl_io_start() which a calls a bunch (how large in that case?) of

6') ->cio_start() instances. Hopefully that'll be vvp_io_write_start() which will pass the value of io->u.ci_rw.crw_pos (picked via an overlapping field of union) to generic_file_aio_write(). Which, BTW, updates ->ki_pos. Then we return into cl_io_loop(), where

4'') we call cl_io_end(), cl_io_unlock() and

5'') cl_io_rw_advance() which increments ....crw_pos, hopefully in sync with what we have in iocb->ki_pos. And calls a bunch of methods. And return into cl_io_loop(), where

4''') we call cl_io_iter_fini() (_another_ pile of methods called) and possibly repeat everything from (4') on (apparently only if nothing had been done so far). Eventually we return into ll_file_io_generic() and there

3''') we copy ....crw_pos into iocb->ki_pos. WTF do we need that? Hadn't generic_file_aio_write() been good enough?

Is that correct? I have *not* traced it into all methods that might've been called in process - stuff called from cl_io_loop() is chock-full of those. Have I missed anything relevant wrt file position handling in there?

You guys really should be forced to hand-draw a call graph for that thing. Both as a punishment and a deterrent...

#4 lore raw
Quoted 10 lines · Al Viro

Egads. Correct me if I'm wrong, please, but it looks like you have

1) ->aio_write() stash its arguments in an off-stack struct and pass it and &iocb->ki_pos to

2) ll_file_io_generic() which calls

3) cl_io_rw_init() which copied ki_pos value passed to it into io->u.ci_rw.crw_pos (in another off-stack struct) and calls

The off-stack storage of per-thread info is used to avoid stack overflow because of the 4kB stack limitation, and to avoid kmalloc/kfree of the same data structures for every call into the callpath. Other code in the kernel handles similar problems by having a separate thread do the lower level IO, but that adds context switches to the IO path. I'm not a big fan of this, but that is what we live with due to limited stack size.

Quoted 9 lines

4) cl_io_init() which calls

5) cl_io_init0() which possibly calls a bunch of instances of ->coo_io_init(),
which may or may not return 0; I _hope_ that in this case that's what'll happen and we get back to ll_file_io_generic(), where

3') we stash iocb and iov/iovlen into the third off-stack structure (cio) and call

The reason for separate IO structures at different parts of the stack is that the Lustre client code has different upper level VFS interfaces for Linux, MacOS, WinNT, and a userspace library. The kernel iocb is not available in all of those cases.

Those non-Linux VFS interfaces are deprecated, and we are looking to remove these abstractions again, but it will take time.

Quoted 21 lines

4') cl_io_loop() where we (in a loop) call cl_io_iter_init(), cl_io_lock() and

5') cl_io_start() which a calls a bunch (how large in that case?) of

6') ->cio_start() instances. Hopefully that'll be vvp_io_write_start() which will pass the value of io->u.ci_rw.crw_pos (picked via an overlapping field of union) to generic_file_aio_write(). Which, BTW, updates ->ki_pos. Then we return into cl_io_loop(), where

4'') we call cl_io_end(), cl_io_unlock() and

5'') cl_io_rw_advance() which increments ....crw_pos, hopefully in sync with what we have in iocb->ki_pos. And calls a bunch of methods. And return into cl_io_loop(), where

4''') we call cl_io_iter_fini() (_another_ pile of methods called) and possibly repeat everything from (4') on (apparently only if nothing had been done so far). Eventually we return into ll_file_io_generic() and there

Quoted 7 lines

3''') we copy ....crw_pos into iocb->ki_pos. WTF do we need that? Hadn't generic_file_aio_write() been good enough?

Is that correct? I have *not* traced it into all methods that might've been called in process - stuff called from cl_io_loop() is chock-full of those. Have I missed anything relevant wrt file position handling in there?

This internal separation of the IO submission path was done to allow parallel IO generation for multiple target devices and parallel RPC generation. Pretty much everyone (except original author, who no longer works on Lustre) agrees that the level of abstraction is too much, and we need to simplify it so that more than a handful of people can even understand it.

Quoted 2 lines

You guys really should be forced to hand-draw a call graph for that thing. Both as a punishment and a deterrent...

Believe me, this is one of my least favourite, and most complex parts of the Lustre code, and we've had many internal discussions and plans to clean it up. Some of that cleanup work has already started, and more is planned, but it will take a while to test it and push it upstream.

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
maintainer
#5 lore raw
Quoted 7 lines · Andreas

The off-stack storage of per-thread info is used to avoid stack overflow because of the 4kB stack limitation, and to avoid kmalloc/kfree of the same data structures for every call into the callpath. Other code in the kernel handles similar problems by having a separate thread do the lower level IO, but that adds context switches to the IO path. I'm not a big fan of this, but that is what we live with due to limited stack size.

<wry> Reducing the depth of call chains also might help. </wry> And yes, I understand the motivation; makes keeping track of what's going on really unpleasant, though...

Quoted 6 lines
Quoted 6 lines

4) cl_io_init() which calls

5) cl_io_init0() which possibly calls a bunch of instances of ->coo_io_init(),
which may or may not return 0; I _hope_ that in this case that's what'll happen and we get back to ll_file_io_generic(), where

Umm... So am I right about the thing returning 0 in this case and in case of ->aio_read()?

Quoted 3 lines
Quoted 3 lines

5') cl_io_start() which a calls a bunch (how large in that case?) of

6') ->cio_start() instances. Hopefully that'll be vvp_io_write_start()

Again, is that assumption correct? I'm not interested in criticizing overall design, etc.; what I am interested in is what does that code end up doing wrt carrying iov/iovlen/iocb/position through the entire thing.

Quoted 4 lines
Quoted 4 lines

which will pass the value of io->u.ci_rw.crw_pos (picked via an overlapping field of union) to generic_file_aio_write(). Which, BTW, updates ->ki_pos. Then we return into cl_io_loop(), where

Quoted 8 lines
Quoted 8 lines

4'') we call cl_io_end(), cl_io_unlock() and

5'') cl_io_rw_advance() which increments ....crw_pos, hopefully in sync with what we have in iocb->ki_pos. And calls a bunch of methods. And return into cl_io_loop(), where

3''') we copy ....crw_pos into iocb->ki_pos. WTF do we need that? Hadn't generic_file_aio_write() been good enough?

Is it safe to assume that
* these two calls of generic_file_aio_{read,write} will always
get pos equal to iocb->ki_pos
* iocb/iov/iovlen will be unchanged since the moment they'd been
passed to ll_file_aio_{read,write}()
* ->ki_pos assigment after the call of cl_io_loop() is no-op for
IO_NORMAL case
* these calls of generic_file_aio_{read,write} will always have
ll_file_aio_{read,write} in the call chain and can't be reached otherwise?

What I want to do is turn ->aio_read()/->aio_write() signature into kiocb x iov_iter -> ssize_t. With all instances, including
generic_file_aio_{read,write}() converted to that. With pos delivered via
iocb and iov/nrsegs via iov_iter. Supplying that in places where they are called via method is trivial and for such calls we definitely have pos == iocb->ki_pos. Most of the places calling those instances directly (not via method) are in other instances of the same methods and pass the arguments unchanged. These two in lustre are, in fact, the only real exceptions. IF the hypotheses above are correct, we can convert those two - just store iov_iter reference instead of iov/nrsegs in vvp_io_args and in ccc_io and that'll do the trick. Probably put iov_iter alongside the vti_local_iovec as well, but I really wonder if that's worth the trouble - might be better to use do_sync_read/do_sync_write and be done with that. Sure, it will slightly increase the stack footprint, but if you are _that_ tight on stack... Anyway, that's a separate story - we can deal with that later.

PS: I'm serious about the call graph, ideally along with the stack footprints of all functions. At least it would give some idea how much do various paths contribute... Of course, it's a real bitch to automate with the amount of indirect calls you've got there - the tricky part is finding the set of functions that might be called at given location ;-/

#6 lore raw

Hello!

Quoted 5 lines · Al Viro

PS: I'm serious about the call graph, ideally along with the stack footprints of all functions. At least it would give some idea how much do various paths contribute... Of course, it's a real bitch to automate with the amount of indirect calls you've got there - the tricky part is finding the set of functions that might be called at given location ;-/

If you are really curious.
Here are the three documents that used to be true soon after the functionality was first implemented.
Since then a bunch of changes (sometimes passed as simplification, sometimes not) were done. But this is the best set of documentation available (other than the code, of course, which is, sadly, not super readable as you have noticed already). And it should give you some nightmares^Widea.

The documents have sample callgraphs and state machines as initially envisioned (before meeting most of hard reality, should I add).

http://wiki.lustre.org/images/1/1d/CLIO.pdf (meat starts at slide 6, statemachines at slide 11)

http://wiki.lustre.org/images/6/66/CLIO-TOI.pdf (somewhat more in-depth version of the above).

http://wiki.lustre.org/images/3/37/CLIO-TOI-notes.pdf - text only version with a
lot of ascii flowcharts and lengthly explanations that was delivered to us when primary designer/developer of the code decided to leave our team.

Bye,
Oleg

#7 lore raw
In reply to Al Viro

On Feb 10, 2014, at 6:40 PM, Al Viro <[email protected]<mailto:[email protected]>> wrote:

On Tue, Feb 11, 2014 at 12:31:39AM +0000, Dilger, Andreas wrote:

The off-stack storage of per-thread info is used to avoid stack overflow because of the 4kB stack limitation, and to avoid kmalloc/kfree of the same data structures for every call into the callpath. Other code in the kernel handles similar problems by having a separate thread do the lower level IO, but that adds context switches to the IO path. I'm not a big fan of this, but that is what we live with due to limited stack size.

<wry> Reducing the depth of call chains also might help. </wry> And yes, I understand the motivation; makes keeping track of what's going on really unpleasant, though...

4) cl_io_init() which calls

5) cl_io_init0() which possibly calls a bunch of instances of ->coo_io_init(),
which may or may not return 0; I _hope_ that in this case that's what'll happen and we get back to ll_file_io_generic(), where

Umm... So am I right about the thing returning 0 in this case and in case of ->aio_read()?

Yes, cio_init() can return one of the following integers:
= 0: IO can go forward

Quoted 1 line

0: IO is not necessary

< 0: error occurs

So it will call aio_read() if cl_io_init() returns zero.

5') cl_io_start() which a calls a bunch (how large in that case?) of

6') ->cio_start() instances. Hopefully that'll be vvp_io_write_start()

Again, is that assumption correct? I'm not interested in criticizing overall design, etc.; what I am interested in is what does that code end up doing wrt carrying iov/iovlen/iocb/position through the entire thing.

Yes, it calls vvp_io_write_start().

Lustre supports data striping, so a file can be composed of multiple OST(Object Storage Target) objects. When reading or writing a striped file, IO is performed stripe by stripe, this is why you can see the loop of cl_io_lock(), cl_io_start(), and cl_io_unlock(), etc.

cl_io_init();
while (still has data) {
cl_io_iter_init();
cl_io_lock();
cl_io_start();
cl_io_end();
cl_io_unlock();
}
cl_io_fini();

Essentially, cl_io_iter_init() separates the IO by stripes, and followed by taking dlm lock, doing actual IO, and release lock, etc.

Let’s take an example here, say a file has two stripes on OST0 and OST1, and stripe size is 1M, so when 2M data is to be written to this file starting from zero, then the above code will loop twice to finish this IO. The first iteration will write [0, 1M) to OST0, and the 2nd one will write [1M, 2M) to OST1. As a result, generic_file_aio_write() will be called twice specifically.

The reason for us to split the IO by stripes is due to cascading problems, which is another story.

which will pass the value of io->u.ci_rw.crw_pos (picked via an overlapping field of union) to generic_file_aio_write(). Which, BTW, updates ->ki_pos. Then we return into cl_io_loop(), where

4'') we call cl_io_end(), cl_io_unlock() and

5'') cl_io_rw_advance() which increments ....crw_pos, hopefully in sync with what we have in iocb->ki_pos. And calls a bunch of methods. And return into cl_io_loop(), where

3''') we copy ....crw_pos into iocb->ki_pos. WTF do we need that? Hadn't generic_file_aio_write() been good enough?

Is it safe to assume that
* these two calls of generic_file_aio_{read,write} will always get pos equal to iocb->ki_pos

Yes.

* iocb/iov/iovlen will be unchanged since the moment they'd been passed to ll_file_aio_{read,write}()

Yes. They will be updated after an IO for one stripe is finished.

* ->ki_pos assigment after the call of cl_io_loop() is no-op for IO_NORMAL case

Yes. But as you have already noticed, the ppos in ll_file_io_generic() is NOT always &iocb->ki_pos. So it isn’t totally wrong to have

“*ppos = io->u.ci_wr.wr.crw_pos;”

in ll_file_io_generic(), take a look at ll_file_splice_read().

* these calls of generic_file_aio_{read,write} will always have ll_file_aio_{read,write} in the call chain and can't be reached otherwise?

Yes.

What I want to do is turn ->aio_read()/->aio_write() signature into kiocb x iov_iter -> ssize_t. With all instances, including
generic_file_aio_{read,write}() converted to that. With pos delivered via
iocb and iov/nrsegs via iov_iter. Supplying that in places where they are called via method is trivial and for such calls we definitely have pos == iocb->ki_pos. Most of the places calling those instances directly

It’s a good to use iov_iter{} to replace iov/nrsegs.

Even this, we still have to remember pos in cl_io{} because the file position will be used at LOV layer to determine the stripe boundary.

(not via method) are in other instances of the same methods and pass the arguments unchanged. These two in lustre are, in fact, the only real exceptions. IF the hypotheses above are correct, we can convert those two - just store iov_iter reference instead of iov/nrsegs in vvp_io_args and in ccc_io and that'll do the trick. Probably put iov_iter alongside the vti_local_iovec as well, but I really wonder if that's worth the trouble - might be better to use do_sync_read/do_sync_write and be done with that.

How do we handle ->splice_read() in this case. We can simplify read and write a little bit, but it seems we have to duplicate some code for splice_read(). Please advise.

Thank you very much for giving the advice.

Jinshan

Sure, it will slightly increase the stack footprint, but if you are _that_ tight on stack... Anyway, that's a separate story - we can deal with that later.

PS: I'm serious about the call graph, ideally along with the stack footprints of all functions. At least it would give some idea how much do various paths contribute... Of course, it's a real bitch to automate with the amount of indirect calls you've got there - the tricky part is finding the set of functions that might be called at given location ;-/

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#8 lore raw
Quoted 10 lines · Jinshan
Quoted 9 lines

4) cl_io_init() which calls

5) cl_io_init0() which possibly calls a bunch of instances of ->coo_io_init(),
which may or may not return 0; I _hope_ that in this case that's what'll happen and we get back to ll_file_io_generic(), where

Umm... So am I right about the thing returning 0 in this case and in case of ->aio_read()?

Quoted 6 lines

Yes, cio_init() can return one of the following integers:
= 0: IO can go forward

Quoted 1 line

0: IO is not necessary

< 0: error occurs

So it will call aio_read() if cl_io_init() returns zero.

Er... The question is the other way round, actually - will it return zero when we arrive there from ll_file_aio_read()? Or could it happen that it returns something positive in that case?

Quoted 2 lines

Let’s take an example here, say a file has two stripes on OST0 and OST1, and stripe size is 1M, so when 2M data is to be written to this file starting from zero, then the above code will loop twice to finish this IO. The first iteration will write [0, 1M) to OST0, and the 2nd one will write [1M, 2M) to OST1. As a result, generic_file_aio_write() will be called twice specifically.

_Twice_? Oh, I see... ccc_io_advance(), right? With iov_iter the whole ->cio_advance() thing would die, AFAICS.

Quoted 5 lines

Yes. But as you have already noticed, the ppos in ll_file_io_generic() is NOT always &iocb->ki_pos. So it isn’t totally wrong to have

“*ppos = io->u.ci_wr.wr.crw_pos;”

in ll_file_io_generic(), take a look at ll_file_splice_read().

->splice_read/->splice_write is another part of that story; eventually, I hope to get rid of those guys, when we get polymorphic iov_iter sorted out. Short version of the story: turn iov_iter into a tagged structure with variants; primitives working with it would check the tag to decide what to do. Think of *BSD struct uio on steroids; AFAIK, Solaris has kept it as well. One of the cases would be iovec-based, with or without splitting the kernel-space iovec into separate case. That's pretty much what *BSD one provides and what iov_iter does right now. Another case: array of <struct page *page, size_t offset, size_t size> triples. ->splice_write() would simply set such an array for all non-empty pipe_buffers and pass it to ->write_iter(). That gives a usable generic implementation, suitable for pretty much all filesystems. _Maybe_ it can even be taught to deal with page stealing, in a way that would allow to kill ->splice_write() as a method - I didn't get to the point where it would be easy to investigate. ->splice_read() ought to use another iov_iter case, where copy_page_to_iter() would grab a reference to page and plain copy_to_iter() would allocate a page if needed and copy over there.

Hell knows; I'm still dealing with preliminary messes and with splice_write side of things. We'll see what set of primitives will it shake down to. So far it's been spread between several fsdevel threads (and some bits of off-list mail when potentially nasty bugs got caught). I hope to get enough of the queue stable enough and large enough to make it worth discussing. Hopefully today or tomorrow there'll be enough meat on it... In any case I'm going to bring it up on LSF/MM.
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
Quoted 13 lines · Al Viro
Quoted 12 lines · Jinshan
Quoted 9 lines

4) cl_io_init() which calls

5) cl_io_init0() which possibly calls a bunch of instances of ->coo_io_init(),
which may or may not return 0; I _hope_ that in this case that's what'll happen and we get back to ll_file_io_generic(), where

Umm... So am I right about the thing returning 0 in this case and in case of ->aio_read()?

Yes, cio_init() can return one of the following integers:
= 0: IO can go forward

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

0: IO is not necessary

< 0: error occurs

So it will call aio_read() if cl_io_init() returns zero.

Er... The question is the other way round, actually - will it return zero when we arrive there from ll_file_aio_read()? Or could it happen that it returns something positive in that case?

Ah sorry, I missed it.

For ->aio_read(), yes mostly it will return zero so that IO can go forward. It won’t return a positive value for read case.

It can also return -ENODATA for HSM(Hierarchical Storage Management) where the data has already been moved to secondary storage so Lustre has to copy it back and then restart the IO.

Quoted 6 lines
Quoted 3 lines

Let’s take an example here, say a file has two stripes on OST0 and OST1, and stripe size is 1M, so when 2M data is to be written to this file starting from zero, then the above code will loop twice to finish this IO. The first iteration will write [0, 1M) to OST0, and the 2nd one will write [1M, 2M) to OST1. As a result, generic_file_aio_write() will be called twice specifically.

_Twice_? Oh, I see... ccc_io_advance(), right? With iov_iter the whole ->cio_advance() thing would die, AFAICS.

Yes, this will simplify it. This is a typically once-it-works-never-put-your-hands-on-it thing. I will create a patch for this.

Quoted 16 lines
Quoted 6 lines

Yes. But as you have already noticed, the ppos in ll_file_io_generic() is NOT always &iocb->ki_pos. So it isn’t totally wrong to have

“*ppos = io->u.ci_wr.wr.crw_pos;”

in ll_file_io_generic(), take a look at ll_file_splice_read().

->splice_read/->splice_write is another part of that story; eventually, I hope to get rid of those guys, when we get polymorphic iov_iter sorted out. Short version of the story: turn iov_iter into a tagged structure with variants; primitives working with it would check the tag to decide what to do. Think of *BSD struct uio on steroids; AFAIK, Solaris has kept it as well. One of the cases would be iovec-based, with or without splitting the kernel-space iovec into separate case. That's pretty much what *BSD one provides and what iov_iter does right now. Another case: array of <struct page *page, size_t offset, size_t size> triples. ->splice_write()

Quoted 8 lines

would simply set such an array for all non-empty pipe_buffers and pass it to ->write_iter(). That gives a usable generic implementation, suitable for pretty much all filesystems. _Maybe_ it can even be taught to deal with page stealing, in a way that would allow to kill ->splice_write() as a method - I didn't get to the point where it would be easy to investigate. ->splice_read() ought to use another iov_iter case, where copy_page_to_iter() would grab a reference to page and plain copy_to_iter() would allocate a page if needed and copy over there.

So essentially, iov_iter{} will be extended to include enough information to perform all kinds of IO operations to operate file data. Therefore, generic_file_aio_{read,write} will become the only two interfaces to initiate file IO operations, with iov_iter{} as one of their parameters.

I will keep an eye on it and once it’s implemented, I will revise lustre implementation correspondingly.

Quoted 8 lines

Hell knows; I'm still dealing with preliminary messes and with splice_write side of things. We'll see what set of primitives will it shake down to. So far it's been spread between several fsdevel threads (and some bits of off-list mail when potentially nasty bugs got caught). I hope to get enough of the queue stable enough and large enough to make it worth discussing. Hopefully today or tomorrow there'll be enough meat on it... In any case I'm going to bring it up on LSF/MM.

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw
In reply to Al Viro
Quoted 19 lines · Al Viro
Quoted 18 lines · Al Viro
Quoted 5 lines · Andreas
Quoted 5 lines
Quoted 5 lines

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag or define a Lustre-specific flag that isn't of use to most/any other filesystems, use (O_NOCTTY|FASYNC) as the new value. These flag are not meaningful for newly-created regular files and should be OK since O_LOV_DELAY_CREATE is only meaningful for new files.

*shrug*

I can live with that; it's a kludge, but it's less broken than that explicit constant - that one is a non-starter, since O_... flag values are arch-dependent.

I have another question about what you are doing there - the games
you are playing with crw_pos. Is there any reason not to have ->ki_pos updated immediately in lustre_generic_file_read()/lustre_generic_file_write()?

Quoted 8 lines
Quoted 4 lines

These two are the only places in the entire tree where
generic_file_aio_{read,write}() does *not* have ppos argument equal to &iocb->ki_pos and I would very much prefer to kill the sucker off.

Ugh... Sorry, I misread that code. Why the devil do you have the pos argument passed to lustre_generic_file_{read,write}() by address, when both proceed to dereference it and pass the value on?

Indeed. This patch will fix the problem, please check the attachment for the patch.

Jinshan

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

From 9a77c75976f60aa2ca76a91b5935cc7020558b34 Mon Sep 17 00:00:00 2001
From: Jinshan Xiong <[email protected]>
Date: Mon, 10 Feb 2014 14:38:21 -0800
Subject: [PATCH] staging/lustre/llite: remove
lustre_generic_file_{read,write}

It looks like lustre_generic_file_{read,write} are a holdover from 2.6.19 where generic_file_aio_read() replaced generic_file_readv() and cross-kernel interoperability was required for some period of time. Lustre has since removed support for those older kernels, but it looks like the wrappers were not deleted at that time. This patch will delete them.

Pass &iocb->ki_pos as the last argument for these functions instead of crw_pos, since this is the convention for other callers. Verify that this is the same as the current crw_pos argument. This code can likely be cleaned up further in a later patch.

---
 drivers/staging/lustre/lustre/llite/vvp_io.c |   29 +++++++++-----------------
 1 file changed, 10 insertions(+), 19 deletions(-)
drivers/staging/lustre/lustre/llite/vvp_io.c +10 -19
index 93cbfbb..aaee8a4 100644
@@ -474,20 +474,6 @@ static void vvp_io_setattr_fini(const struct lu_env *env,
 	vvp_io_fini(env, ios);
 }
 
-static ssize_t lustre_generic_file_read(struct file *file,
-					struct ccc_io *vio, loff_t *ppos)
-{
-	return generic_file_aio_read(vio->cui_iocb, vio->cui_iov,
-				     vio->cui_nrsegs, *ppos);
-}
-
-static ssize_t lustre_generic_file_write(struct file *file,
-					struct ccc_io *vio, loff_t *ppos)
-{
-	return generic_file_aio_write(vio->cui_iocb, vio->cui_iov,
-				      vio->cui_nrsegs, *ppos);
-}
-
 static int vvp_io_read_start(const struct lu_env *env,
 			     const struct cl_io_slice *ios)
 {
@@ -506,6 +492,7 @@ static int vvp_io_read_start(const struct lu_env *env,
 	int     exceed = 0;
 
 	CLOBINVRNT(env, obj, ccc_object_invariant(obj));
+	LASSERT(cio->cui_iocb->ki_pos == pos);
 
 	CDEBUG(D_VFSTRACE, "read: -> [%lli, %lli)\n", pos, pos + cnt);
 
@@ -540,8 +527,10 @@ static int vvp_io_read_start(const struct lu_env *env,
 	file_accessed(file);
 	switch (vio->cui_io_subtype) {
 	case IO_NORMAL:
-		 result = lustre_generic_file_read(file, cio, &pos);
-		 break;
+		result = generic_file_aio_read(cio->cui_iocb,
+					       cio->cui_iov, cio->cui_nrsegs,
+					       cio->cui_iocb->ki_pos);
+		break;
 	case IO_SPLICE:
 		result = generic_file_splice_read(file, &pos,
 				vio->u.splice.cui_pipe, cnt,
@@ -586,7 +575,6 @@ static int vvp_io_write_start(const struct lu_env *env,
 	struct cl_io       *io    = ios->cis_io;
 	struct cl_object   *obj   = io->ci_obj;
 	struct inode       *inode = ccc_object_inode(obj);
-	struct file	*file  = cio->cui_fd->fd_file;
 	ssize_t result = 0;
 	loff_t pos = io->u.ci_wr.wr.crw_pos;
 	size_t cnt = io->u.ci_wr.wr.crw_count;
@@ -601,6 +589,8 @@ static int vvp_io_write_start(const struct lu_env *env,
 		 */
 		pos = io->u.ci_wr.wr.crw_pos = i_size_read(inode);
 		cio->cui_iocb->ki_pos = pos;
+	} else {
+		LASSERT(cio->cui_iocb->ki_pos == pos);
 	}
 
 	CDEBUG(D_VFSTRACE, "write: [%lli, %lli)\n", pos, pos + (long long)cnt);
@@ -608,8 +598,9 @@ static int vvp_io_write_start(const struct lu_env *env,
 	if (cio->cui_iov == NULL) /* from a temp io in ll_cl_init(). */
 		result = 0;
 	else
-		result = lustre_generic_file_write(file, cio, &pos);
-
+		result = generic_file_aio_write(cio->cui_iocb,
+						cio->cui_iov, cio->cui_nrsegs,
+						cio->cui_iocb->ki_pos);
 	if (result > 0) {
 		if (result < cnt)
 			io->ci_continue = 0;
Signature · 1 line
1.7.9.5
#11 lore raw
In reply to Al Viro
Quoted 7 lines · Al Viro
Quoted 5 lines · Andreas
Quoted 5 lines
Quoted 5 lines

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag or define a Lustre-specific flag that isn't of use to most/any other filesystems, use (O_NOCTTY|FASYNC) as the new value. These flag are not meaningful for newly-created regular files and should be OK since O_LOV_DELAY_CREATE is only meaningful for new files.

Quoted 5 lines

*shrug*

I can live with that; it's a kludge, but it's less broken than that explicit constant - that one is a non-starter, since O_... flag values are arch-dependent.

Greg,
could you please merge the original patch. We'd like to get this into our pending release of the Lustre user tools and into the releases for older kernels (which will support both the old and new flags until the support for older kernels is removed).

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
#12 lore raw
Quoted 14 lines · Andreas
Quoted 7 lines · Al Viro
Quoted 5 lines · Andreas
Quoted 5 lines
Quoted 5 lines

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag or define a Lustre-specific flag that isn't of use to most/any other filesystems, use (O_NOCTTY|FASYNC) as the new value. These flag are not meaningful for newly-created regular files and should be OK since O_LOV_DELAY_CREATE is only meaningful for new files.

Quoted 5 lines

*shrug*

I can live with that; it's a kludge, but it's less broken than that explicit constant - that one is a non-starter, since O_... flag values are arch-dependent.

Quoted 5 lines

Greg,
could you please merge the original patch. We'd like to get this into our pending release of the Lustre user tools and into the releases for older kernels (which will support both the old and new flags until the support for older kernels is removed).

Can you please resend the original patch, without the "RFC" line in the subject, so I know to apply it now?

thanks,

greg k-h

#13 lore raw
In reply to Al Viro
Quoted 3 lines · Al Viro

I can live with that; it's a kludge, but it's less broken than that explicit constant - that one is a non-starter, since O_... flag values are arch-dependent.

Grabbing their own O_FLAG is of course not acceptable at all. Personally I don't think this version is acceptable for real mainline either. What exactly are the semantics of the flag? Why don't you do object allocation on demand like all delalloc filesystems by default?

#14 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Al Viro

I can live with that; it's a kludge, but it's less broken than that explicit constant - that one is a non-starter, since O_... flag values are arch-dependent.

Grabbing their own O_FLAG is of course not acceptable at all. Personally I don't think this version is acceptable for real mainline either. What exactly are the semantics of the flag? Why don't you do object allocation on demand like all delalloc filesystems by default?

This was described in the original patch and follow-on email, but I'll repeat it here, and expand the detail a bit further:

In kernel 3.11 O_TMPFILE was introduced, but the open flag value conflicts with the O_LOV_DELAY_CREATE flag 020000000 previously used by Lustre-aware applications. O_LOV_DELAY_CREATE allows applications to defer file layout and object creation from open time (the default) until it can instead be specified by the application using an ioctl.

The main goal of the O_LOV_DELAY_CREATE flag is to allow the file to be opened in a "preliminary" manner to allow the application to specify the layout of the file across the Lustre storage targets (e.g. whether the app has millions of separate files each one written to a single server, or there is a single huge file spread across all of the servers, or some combination of the two, if it is RAID-0 or RAID-1, or whatever).

FYI, an "object" in Lustre is not a fixed-size chunk of space like Ceph or HDFS that needs to be continuously allocated as a file grows, but rather a variable-sized inode-without-a-name that is written at arbitrary byte offsets and can be sparse, so there is no need for the client and metadata server to communicate after the initial file layout has been decided.

The Lustre object(s) are normally allocated by the metadata server at open time to avoid RPC round-trips and lock contention for files opened by large numbers of nodes at once. The layout is normally specified by the filesystem default, or on the parent directory, but some applications need fine-grained control over the layout to optimize for a particular filesystem configuration.

Instead of trying to find a non-conflicting O_LOV_DELAY_CREATE flag or define a Lustre-specific flag that isn't of use to most/any other filesystems, use (O_NOCTTY|FASYNC) as the new value. These flag are not meaningful for newly-created regular files and should be OK since O_LOV_DELAY_CREATE is only meaningful for new files.

I looked into using O_ACCMODE/FMODE_WRITE_IOCTL, which allows calling ioctl() on the minimally-opened fd and is close to what is needed, but that doesn't allow specifying the actual read or write mode for the file, and fcntl(F_SETFL) doesn't allow O_RDONLY/O_WRONLY/O_RDWR to be set after the file is opened.

We want to avoid the need to have lots of syscalls to do this, since they translate into extra RPCs that we want to avoid when creating potentially millions of files over the network.

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division