Skip to content

RFC ceph: add asynchronous create functionality

Jeff Layton ceph-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest9 patches 18 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9ceph: ensure we have a new cap before continuing in fill_inodefs/ceph/inode.c+4 -10 replies
2/9ceph: print name of xattr being set in set/getxattr dout messagefs/ceph/xattr.c+3 -20 replies
3/9ceph: close some holes in struct ceph_mds_requestfs/ceph/mds_client.h+1 -10 replies
4/9ceph: make ceph_fill_inode non-staticfs/ceph/inode.c fs/ceph/super.h+32 -230 replies
5/9libceph: export ceph_file_layout_is_validnet/ceph/ceph_fs.c+1 -00 replies
6/9ceph: decode interval_sets for delegated inosfs/ceph/mds_client.c fs/ceph/mds_client.h+102 -100 replies
7/9ceph: add flag to delegate an inode number for async createfs/ceph/inode.c fs/ceph/mds_client.c fs/ceph/mds_client.h+21 -13 replies
8/9ceph: copy layout, max_size and truncate_size on successful sync createfs/ceph/caps.c fs/ceph/file.c fs/ceph/inode.c+45 -77 replies
9/9ceph: attempt to do async create when possiblefs/ceph/caps.c fs/ceph/file.c fs/ceph/mds_client.c +3 more+191 -197 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/9] ceph: add asynchronous create functionality

I recently sent a patchset that allows the client to do an asynchronous UNLINK call to the MDS when it has the appropriate caps and dentry info. This set adds the corresponding functionality for creates.

When the client has the appropriate caps on the parent directory and dentry information, and a delegated inode number, it can satisfy a request locally without contacting the server. This allows the kernel client to return very quickly from an O_CREAT open, so it can get on with doing other things.

These numbers are based on my personal test rig, which is a KVM client vs a vstart cluster running on my workstation (nothing scientific here).

A simple benchmark (with the cephfs mounted at /mnt/cephfs):
-------------------8<-------------------
#!/bin/sh

TESTDIR=/mnt/cephfs/test-dirops.$$

mkdir $TESTDIR
stat $TESTDIR
echo "Creating files in $TESTDIR"
time for i in seq 1 10000; do
echo "foobarbaz" > $TESTDIR/$i
done
-------------------8<-------------------

With async dirops disabled:

real 0m9.865s
user 0m0.353s
sys 0m0.888s

With async dirops enabled:

real 0m5.272s
user 0m0.104s
sys 0m0.454s

That workload is a bit synthetic though. One workload we're interested in improving is untar. Untarring a deep directory tree (random kernel tarball I had laying around):

Disabled:

$ time tar xf ~/linux-4.18.0-153.el8.jlayton.006.tar

real 1m35.774s
user 0m0.835s
sys 0m7.410s

Enabled:

$ time tar xf ~/linux-4.18.0-153.el8.jlayton.006.tar

real 1m32.182s
user 0m0.783s
sys 0m6.830s

Not a huge win there. I suspect at this point that synchronous mkdir may be serializing behind the async creates.

It needs a lot more performance tuning and analysis, but it's now at the point where it's basically usable. To enable it, turn on the ceph.enable_async_dirops module option.

There are some places that need further work:

1) The MDS patchset to delegate inodes to the client is not yet merged:

    https://github.com/ceph/ceph/pull/31817

2) this is 64-bit arch only for the moment. I'm using an xarray to track the delegated inode numbers, and those don't do 64-bit indexes on 32-bit machines. Is anyone using 32-bit ceph clients? We could probably build an xarray of xarrays if needed.

3) The error handling is still pretty lame. If the create fails, it'll set a writeback error on the parent dir and the inode itself, but the client could end up writing a bunch before it notices, if it even bothers to check. We probably need to do better here. I'm open to suggestions on this bit especially.

Jeff Layton (9):
ceph: ensure we have a new cap before continuing in fill_inode
ceph: print name of xattr being set in set/getxattr dout message
ceph: close some holes in struct ceph_mds_request
ceph: make ceph_fill_inode non-static
libceph: export ceph_file_layout_is_valid
ceph: decode interval_sets for delegated inos
ceph: add flag to delegate an inode number for async create
ceph: copy layout, max_size and truncate_size on successful sync
create
ceph: attempt to do async create when possible

 fs/ceph/caps.c               |  31 +++++-
 fs/ceph/file.c               | 202 +++++++++++++++++++++++++++++++++--
 fs/ceph/inode.c              |  57 +++++-----
 fs/ceph/mds_client.c         | 130 ++++++++++++++++++++--
 fs/ceph/mds_client.h         |  12 ++-
 fs/ceph/super.h              |  10 ++
 fs/ceph/xattr.c              |   5 +-
 include/linux/ceph/ceph_fs.h |   8 +-
 net/ceph/ceph_fs.c           |   1 +
 9 files changed, 396 insertions(+), 60 deletions(-)
Signature · 1 line
2.24.1