lustre: update to 2.11 support
Subsystems (2)
cover lustre: update to 2.11 support
This patchset covers 3 areas. The fix 6 patches resolves critical bugs in the linux client. Mainly for handling of LNet with secondary IP aliases as well as an RCU issue reported with debugging enabled. This consoldation of IP handling makes any kernel API changes in the future easier to handle.
The last 9 patches cover patches in the lustre-testing that landed to the OpenSFS branch.
The rest of the patches cover all the patches missing from the Lustre 2.11 release. I have done extentsive testing of these patches (sanity, sanity-pfl, sanity-dom, sanity-pfl). Only regression I saw was sanity-flr 200 which needs to be investiaged. So we are in very good shape.
Alex Zhuravlev (1):
lustre: support for gcc8
Alexander Boyko (2):
lustre: hsm: don't release with wrong size
lustre: obd: add check to obd_statfs
Amir Shehata (8):
lnet: fix memory leak and lnet_interfaces_max
lnet: decref on peer after use
lnet: rediscover peer if it changed
lnet: resolve unsafe list access
lnet: don't discover loopback interface
lnet: reduce logging severity
lnet: safe access in debug print
lnet: reduce discovery timeout
Andreas Dilger (11):
lustre: seq: make seq_proc_write_common() safer
lustre: llog: consolidate common error checking
lustre: llite: increase whole-file readahead to RPC size
lustre: ldlm: remove liblustre remnants
lustre: misc: replace LASSERT() with BUILD_BUG_ON()
lustre: lov: allow lov.*.stripe{size,count}=-1 param
lustre: idl: clean up and document ptlrpc structures
lustre: idl: remove obsolete RPC MSG flags
lustre: misc: update Intel copyright messages for 2017
lustre: idl: add PTLRPC definitions to enum
lustre: uapi: rename LCM_FL_NOT_FLR to LCM_FL_NONE
Andriy Skulysh (1):
lustre: llite: NULL pointer dereference in cl_object_top()
Ben Evans (2):
lustre: jobstats: move jobstats code into separate file.
lustre: llite: Add exit for filedata allocation failed
Bobi Jam (4):
lustre: flr: instantiate component for truncate
lustre: flr: split a mirror from mirrored file
lustre: flr: layout truncate compatibility
lustre: osc: add a bit to indicate osc_page in cache tree
Bruno Faccini (5):
lnet: Allocate MEs and small MDs in own kmem_caches
lustre: obd: free obd_svc_stats when all users are gone
lustre: llite: avoid live-lock when concurrent mmap()s
lustre: ptlrpc: allow to limit number of service's rqbds
lnet: ensure peer put back on dc request queue
Dmitry Eremin (1):
lnet: Fix lost lock
Dominique Martinet (1):
lustre: lov: fill no-extent fiemap on object with no stripe.
Emoly Liu (1):
lustre: fid: improve seq allocation error messages
Erich Focht (1):
lustre: llite: enable readahead for small read_ahead_per_file
Fan Yong (3):
lustre: scrub: general framework for OI scrub
lustre: mdc: interruptable during RPC retry for EINPROGRESS
lustre: acl: prepare small buffer for ACL RPC reply
Gregoire Pichon (1):
lustre: ldlm: testcases for multiple modify RPCs feature
Hongchao Zhang (1):
lustre: ldlm: fix export reference
James Simmons (20):
lnet: fix needed headers for lnet headers
lnet: ksocklnd: add secondary IP address handling
lnet: o2iblnd: add secondary IP address handling
lnet: consoldate secondary IP address handling
lustre: all: remove all Sun license and URL references
lustre: ptlrpc: change cr_sent_tv from timespec to ktime
lustre: ldlm: don't use jiffies as sysfs parameter
lustre: ptlrpc: migrate pinger to 64 bit time
lustre: ldlm: migrate the rest of the code to 64 bit time
lustre: llite: sync bdi sysfs name with lustre sysfs tree
lnet: libcfs: call proper crypto algo when keys are passed in
lustre: sec: migrate to 64 bit time
lustre: llite: change lli_glimpse_time to ktime
lustre: osc: migrate to 64 bit time
lustre: libcfs: remove cfs_time_XXX_64 wrappers
lustre: address issues raised by gcc7
lustre: obd: remove s2dhms time function
lustre: obd: fix statfs handling
lustre: update version to 2.10.99
lustre: ptlrpc: clarify 64 bit time usage
Jeremy Filizetti (1):
lustre: ldlm: Don't check opcode with NULL rq_reqmsg
Jinshan Xiong (17):
lustre: lov: take lov layout lock for I/O with ignore_layout
lustre: lov: pack lsm_flags from layout
lustre: clio: introduce CIT_GLIMPSE for glimpse
lustre: flr: add infrastructure to create a new mirror
lustre: clio: no glimpse for data immutable file
lustre: flr: read support for flr
lustre: lov: rework write intent on componect instantiation
lustre: ptlrpc: use lu_extent in layout_intent
lustre: flr: Send write intent RPC to mdt
lustre: flr: extend DATA_VERSION API to read layout version
lustre: lov: skip empty pages in lov_io_submit()
lustre: flr: mirror read and write
lustre: flr: resync support and test tool
lustre: flr: randomize mirror pick
lustre: flr: comp-flags support when creating mirrors
lustre: flr: revise lease API
lustre: llite: deny 2.10 clients to open mirrored files
John L. Hammond (5):
lustre: ldlm: assume OBD_CONNECT_IBITS
lustre: llite: assume OBD_CONNECT_ATTRFID
lustre: llite: simplify ll_inode_revalidate()
lustre: clio: remove unused cl_lock layers
lustre: hsm: filter kkuc write by client UUID
Kit Westneat (1):
lnet: fix contiguous range support
Lai Siyao (6):
lustre: statahead: support striped directory
lustre: readdir: improve striped readdir
lustre: lmv: stripe dir page may be released mistakenly
lustre: mdc: don't assert on name pack
lustre: dne: allow mkdir with specific MDTs
lustre: recovery: support setstripe replay
Li Wei (1):
lustre: ptlrpc: Fix an rq_no_reply assertion failure
Matthew Sanderson (1):
lustre: mdc: Add an additional set of 64 changelog flags
Mikhail Pershin (1):
lustre: dom: support DATA_VERSION IO type
Mikhal Pershin (15):
lustre: osc: prepare OSC code to be used from MDC
lustre: mdc: add cl_device to the MDC
lustre: lov: add MDT target to the LOV device
lustre: mdt: IO request handling in MDT
lustre: osc: common client setup/cleanup
lustre: mdc: add IO methods to the MDC
lustre: lvbo: pass lock as parameter to lvbo_update()
lustre: mds: add IO locking to the MDC and MDT
lustre: mdc: add IO stats in mdc
lustre: lov: add Data-on-MDT tests and fixes
lustre: mdc: use generic grant code at MDT
lustre: mds: combine DoM bit with other IBITS
lustre: llite: check layout size after cl_object_layout_get
lustre: mdc: implement own mdc_io_fsync_start()
lustre: mdc: high-priority request handling for DOM
NeilBrown (13):
lustre: fix signal handling in abortable waits.
lustre: ptlrpc: add watchdog for ptlrpc service threads.
lustre: handles: discard h_owner in favour of h_ops
lustre: ldlm: Remove use of SLAB_DESTROY_BY_RCU for ldlm lock slab
lustre: ldlm: simplify lock_mode_to_index()
lustre: ptlrpc: use list_move where appropriate.
lustre: ptlrpc: simplify locking in ptlrpc_add_rqs_to_pool()
lustre: ptlrpc: incorporate BUILD_BUG_ON into ptlrpc_req_async_args()
lustre: introduce CONFIG_LUSTRE_FS_POSIX_ACL
lustre: ptlrpc: discard a server-only waitq.
lustre: llite: remove // comments.
lustre: remove outdated comments about ->ap_* functions.
lustre: clean up some comment alignment.
Olaf Weber (1):
lnet: Handle ping buffer with only loopback NID
Patrick Farrell (8):
lustre: ldlm: Use interval tree to update kms
lustre: llite: Reduce overhead for ll_do_fast_read
lustre: llite: Implement ladvise lockahead
lustre: brw: add short io osc/ost transfer.
lustre: vvp: Print discarded page warning on -EIO
lustre: clio: Use readahead for partial page write
lustre: llite: Add tiny write support
lustre: llite: Disable tiny writes for append
Qian Yingjin (1):
lustre: misc: Wrong checksum return value
Sebastien Buisson (8):
lustre: ptlrpc: remove incorrect pid printing
lustre: mdc: add uid/gid to Changelogs entries
lustre: lustre: move LA_* flags to lustre_user.h
lustre: mdc: add client NID to Changelogs entries
lustre: mdc: implement CL_OPEN for Changelogs
lustre: mdc: add CL_GETXATTR for Changelogs
lustre: uapi: record denied OPEN in Changelogs
lustre: uapi: replace FMODE_{READ,WRITE} with MDS_* equivs
Sonia Sharma (1):
lnet: Remove LASSERT on userspace data
Steve Guminski (2):
lustre: ptlrpc: Use C99 initializer in ptlrpc_register_rqbd()
lnet: selftest: Use C99 struct initializer in framework.c
Vladimir Saveliev (2):
lustre: llite: have ll_write_end to sync for DIO
lustre: llite: fix mount error handing
Wang Di (4):
lustre: fld: resend seq lookup RPC if it is on LWP
lustre: fld: retry fld rpc even for ESHUTDOWN
lustre: fld: retry fld rpc until the import is closed
lustre: fld: fld client lookup should retry
Documentation/lustre/ladvise_lockahead.txt | 304 ++++
fs/lustre/Kconfig | 11 +
fs/lustre/fid/fid_internal.h | 2 +-
fs/lustre/fid/fid_request.c | 12 +-
fs/lustre/fid/lproc_fid.c | 14 +-
fs/lustre/fld/fld_request.c | 42 +-
fs/lustre/include/cl_object.h | 106 +-
fs/lustre/include/lprocfs_status.h | 28 +-
fs/lustre/include/lu_object.h | 25 +-
fs/lustre/include/lu_ref.h | 2 +-
fs/lustre/include/lustre_acl.h | 8 +-
fs/lustre/include/lustre_disk.h | 8 +-
fs/lustre/include/lustre_dlm.h | 88 +-
fs/lustre/include/lustre_dlm_flags.h | 54 +-
fs/lustre/include/lustre_export.h | 37 +-
fs/lustre/include/lustre_fid.h | 9 +-
fs/lustre/include/lustre_handles.h | 9 +-
fs/lustre/include/lustre_import.h | 28 +-
fs/lustre/include/lustre_kernelcomm.h | 14 +-
fs/lustre/include/lustre_lib.h | 8 +-
fs/lustre/include/lustre_linkea.h | 2 +-
fs/lustre/include/lustre_lmv.h | 2 +-
fs/lustre/include/lustre_log.h | 2 +-
fs/lustre/include/lustre_mdc.h | 2 +-
fs/lustre/include/lustre_mds.h | 7 +-
fs/lustre/include/lustre_net.h | 91 +-
.../osc_cl_internal.h => include/lustre_osc.h} | 440 +++++-
fs/lustre/include/lustre_req_layout.h | 8 +-
fs/lustre/include/lustre_sec.h | 4 +-
fs/lustre/include/lustre_swab.h | 3 +-
fs/lustre/include/obd.h | 62 +-
fs/lustre/include/obd_class.h | 45 +-
fs/lustre/include/obd_support.h | 13 +-
fs/lustre/ldlm/ldlm_extent.c | 71 +-
fs/lustre/ldlm/ldlm_flock.c | 2 +-
fs/lustre/ldlm/ldlm_inodebits.c | 22 +
fs/lustre/ldlm/ldlm_internal.h | 9 +-
fs/lustre/ldlm/ldlm_lib.c | 18 +-
fs/lustre/ldlm/ldlm_lock.c | 30 +-
fs/lustre/ldlm/ldlm_lockd.c | 14 +-
fs/lustre/ldlm/ldlm_plain.c | 2 +-
fs/lustre/ldlm/ldlm_pool.c | 6 +-
fs/lustre/ldlm/ldlm_request.c | 90 +-
fs/lustre/ldlm/ldlm_resource.c | 31 +-
fs/lustre/llite/Makefile | 4 +-
fs/lustre/llite/dcache.c | 2 +-
fs/lustre/llite/dir.c | 25 +-
fs/lustre/llite/file.c | 1104 +++++++++++----
fs/lustre/llite/glimpse.c | 58 +-
fs/lustre/llite/lcommon_cl.c | 12 +-
fs/lustre/llite/lcommon_misc.c | 5 +-
fs/lustre/llite/llite_internal.h | 60 +-
fs/lustre/llite/llite_lib.c | 104 +-
fs/lustre/llite/llite_mmap.c | 14 +-
fs/lustre/llite/llite_nfs.c | 2 +-
fs/lustre/llite/lproc_llite.c | 62 +-
fs/lustre/llite/namei.c | 61 +-
fs/lustre/llite/rw.c | 98 +-
fs/lustre/llite/rw26.c | 157 ++-
fs/lustre/llite/statahead.c | 14 +-
fs/lustre/llite/vvp_dev.c | 6 -
fs/lustre/llite/vvp_internal.h | 15 +-
fs/lustre/llite/vvp_io.c | 89 +-
fs/lustre/llite/vvp_lock.c | 87 --
fs/lustre/llite/vvp_object.c | 10 +-
fs/lustre/llite/vvp_page.c | 15 +-
fs/lustre/llite/xattr.c | 6 +-
fs/lustre/llite/xattr_cache.c | 2 +-
fs/lustre/lmv/lmv_intent.c | 2 +-
fs/lustre/lmv/lmv_internal.h | 2 +-
fs/lustre/lmv/lmv_obd.c | 538 +++----
fs/lustre/lmv/lproc_lmv.c | 2 +-
fs/lustre/lov/Makefile | 2 +-
fs/lustre/lov/lov_cl_internal.h | 273 +++-
fs/lustre/lov/lov_dev.c | 284 +++-
fs/lustre/lov/lov_ea.c | 24 +-
fs/lustre/lov/lov_internal.h | 13 +-
fs/lustre/lov/lov_io.c | 490 ++++++-
fs/lustre/lov/lov_lock.c | 17 +-
fs/lustre/lov/lov_merge.c | 2 +-
fs/lustre/lov/lov_obd.c | 45 +-
fs/lustre/lov/lov_object.c | 842 +++++++----
fs/lustre/lov/lov_offset.c | 5 +-
fs/lustre/lov/lov_pack.c | 4 +-
fs/lustre/lov/lov_page.c | 17 +-
fs/lustre/lov/lov_pool.c | 2 +-
fs/lustre/lov/lov_request.c | 4 +-
fs/lustre/lov/lovsub_lock.c | 81 --
fs/lustre/lov/lovsub_object.c | 4 +-
fs/lustre/lov/lproc_lov.c | 4 +-
fs/lustre/mdc/Makefile | 1 +
fs/lustre/mdc/lproc_mdc.c | 320 ++++-
fs/lustre/mdc/mdc_changelog.c | 11 +-
fs/lustre/mdc/mdc_dev.c | 1479 ++++++++++++++++++++
fs/lustre/mdc/mdc_internal.h | 12 +-
fs/lustre/mdc/mdc_lib.c | 45 +-
fs/lustre/mdc/mdc_locks.c | 130 +-
fs/lustre/mdc/mdc_reint.c | 78 +-
fs/lustre/mdc/mdc_request.c | 206 ++-
fs/lustre/mgc/lproc_mgc.c | 2 +-
fs/lustre/mgc/mgc_internal.h | 2 +-
fs/lustre/mgc/mgc_request.c | 4 +-
fs/lustre/obdclass/Makefile | 3 +-
fs/lustre/obdclass/cl_internal.h | 2 +-
fs/lustre/obdclass/cl_io.c | 18 +-
fs/lustre/obdclass/cl_lock.c | 11 +-
fs/lustre/obdclass/cl_object.c | 14 +-
fs/lustre/obdclass/cl_page.c | 15 +-
fs/lustre/obdclass/class_obd.c | 45 +-
fs/lustre/obdclass/genops.c | 36 +-
fs/lustre/obdclass/jobid.c | 86 ++
fs/lustre/obdclass/kernelcomm.c | 28 +-
fs/lustre/obdclass/linkea.c | 2 +-
fs/lustre/obdclass/llog.c | 4 +-
fs/lustre/obdclass/llog_cat.c | 50 +-
fs/lustre/obdclass/llog_internal.h | 2 +-
fs/lustre/obdclass/llog_swab.c | 2 +-
fs/lustre/obdclass/lprocfs_status.c | 131 +-
fs/lustre/obdclass/lu_object.c | 2 +-
fs/lustre/obdclass/lu_ref.c | 2 +-
fs/lustre/obdclass/lustre_handles.c | 8 +-
fs/lustre/obdclass/lustre_peer.c | 2 +-
fs/lustre/obdclass/obd_config.c | 44 +-
fs/lustre/obdclass/obd_mount.c | 2 +-
fs/lustre/obdclass/obd_sysfs.c | 2 +-
fs/lustre/obdclass/obdo.c | 2 +-
fs/lustre/obdecho/debug.c | 2 +-
fs/lustre/obdecho/echo_client.c | 4 +-
fs/lustre/osc/lproc_osc.c | 11 +-
fs/lustre/osc/osc_cache.c | 61 +-
fs/lustre/osc/osc_dev.c | 40 +-
fs/lustre/osc/osc_internal.h | 168 +--
fs/lustre/osc/osc_io.c | 151 +-
fs/lustre/osc/osc_lock.c | 190 +--
fs/lustre/osc/osc_object.c | 90 +-
fs/lustre/osc/osc_page.c | 97 +-
fs/lustre/osc/osc_quota.c | 4 +-
fs/lustre/osc/osc_request.c | 453 +++---
fs/lustre/ptlrpc/client.c | 56 +-
fs/lustre/ptlrpc/events.c | 2 +-
fs/lustre/ptlrpc/import.c | 87 +-
fs/lustre/ptlrpc/layout.c | 46 +-
fs/lustre/ptlrpc/llog_net.c | 2 +-
fs/lustre/ptlrpc/lproc_ptlrpc.c | 68 +-
fs/lustre/ptlrpc/niobuf.c | 15 +-
fs/lustre/ptlrpc/pack_generic.c | 125 +-
fs/lustre/ptlrpc/pinger.c | 44 +-
fs/lustre/ptlrpc/ptlrpc_internal.h | 4 +-
fs/lustre/ptlrpc/ptlrpc_module.c | 2 +-
fs/lustre/ptlrpc/ptlrpcd.c | 4 +-
fs/lustre/ptlrpc/recover.c | 2 +-
fs/lustre/ptlrpc/sec.c | 3 +-
fs/lustre/ptlrpc/sec_bulk.c | 24 +-
fs/lustre/ptlrpc/sec_config.c | 2 +-
fs/lustre/ptlrpc/sec_gc.c | 2 +-
fs/lustre/ptlrpc/sec_null.c | 6 +-
fs/lustre/ptlrpc/sec_plain.c | 8 +-
fs/lustre/ptlrpc/service.c | 180 ++-
fs/lustre/ptlrpc/wiretest.c | 271 +++-
include/linux/libcfs/libcfs.h | 2 +-
include/linux/libcfs/libcfs_cpu.h | 2 +-
include/linux/libcfs/libcfs_debug.h | 7 +-
include/linux/libcfs/libcfs_fail.h | 1 +
include/linux/libcfs/libcfs_hash.h | 2 +-
include/linux/libcfs/libcfs_private.h | 5 +-
include/linux/libcfs/libcfs_string.h | 2 +-
include/linux/lnet/api.h | 2 +-
include/linux/lnet/lib-lnet.h | 52 +-
include/linux/lnet/lib-types.h | 16 +-
include/uapi/linux/lnet/libcfs_debug.h | 2 +-
include/uapi/linux/lnet/libcfs_ioctl.h | 2 +
include/uapi/linux/lnet/lnet-dlc.h | 2 +-
include/uapi/linux/lnet/lnet-types.h | 2 +-
include/uapi/linux/lnet/lnetctl.h | 3 +
include/uapi/linux/lnet/lnetst.h | 2 +
include/uapi/linux/lnet/nidstr.h | 10 +-
include/uapi/linux/lustre/lustre_fid.h | 2 +-
include/uapi/linux/lustre/lustre_idl.h | 384 ++---
include/uapi/linux/lustre/lustre_ioctl.h | 2 +-
include/uapi/linux/lustre/lustre_ostid.h | 2 +-
include/uapi/linux/lustre/lustre_param.h | 28 +-
include/uapi/linux/lustre/lustre_user.h | 608 +++++++-
include/uapi/linux/lustre/lustre_ver.h | 4 +-
net/lnet/klnds/o2iblnd/o2iblnd.c | 173 +--
net/lnet/klnds/o2iblnd/o2iblnd.h | 80 +-
net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 89 +-
net/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 2 +-
net/lnet/klnds/socklnd/socklnd.c | 140 +-
net/lnet/klnds/socklnd/socklnd.h | 2 +-
net/lnet/klnds/socklnd/socklnd_cb.c | 28 +-
net/lnet/klnds/socklnd/socklnd_lib.c | 10 +-
net/lnet/klnds/socklnd/socklnd_proto.c | 2 +-
net/lnet/libcfs/debug.c | 11 +-
net/lnet/libcfs/libcfs_cpu.c | 2 +-
net/lnet/libcfs/libcfs_string.c | 2 +-
net/lnet/libcfs/linux-crypto.c | 31 +-
net/lnet/libcfs/module.c | 14 +-
net/lnet/libcfs/tracefile.c | 10 +-
net/lnet/libcfs/tracefile.h | 2 +-
net/lnet/lnet/acceptor.c | 8 +-
net/lnet/lnet/api-ni.c | 141 +-
net/lnet/lnet/config.c | 129 +-
net/lnet/lnet/lib-md.c | 4 +-
net/lnet/lnet/lib-me.c | 2 +-
net/lnet/lnet/lib-move.c | 38 +-
net/lnet/lnet/lib-msg.c | 2 +-
net/lnet/lnet/lib-ptl.c | 4 +-
net/lnet/lnet/lib-socket.c | 2 +-
net/lnet/lnet/lo.c | 4 +-
net/lnet/lnet/module.c | 2 +-
net/lnet/lnet/net_fault.c | 2 +-
net/lnet/lnet/nidstrings.c | 326 ++---
net/lnet/lnet/peer.c | 128 +-
net/lnet/lnet/router.c | 10 +-
net/lnet/lnet/router_proc.c | 2 +-
net/lnet/selftest/brw_test.c | 2 +-
net/lnet/selftest/conrpc.c | 2 +-
net/lnet/selftest/conrpc.h | 16 +-
net/lnet/selftest/console.h | 96 +-
net/lnet/selftest/framework.c | 58 +-
net/lnet/selftest/rpc.c | 2 +-
net/lnet/selftest/rpc.h | 28 +-
net/lnet/selftest/selftest.h | 40 +-
223 files changed, 10237 insertions(+), 4034 deletions(-)
create mode 100644 Documentation/lustre/ladvise_lockahead.txt
rename fs/lustre/{osc/osc_cl_internal.h => include/lustre_osc.h} (61%)
delete mode 100644 fs/lustre/llite/vvp_lock.c
delete mode 100644 fs/lustre/lov/lovsub_lock.c
create mode 100644 fs/lustre/mdc/mdc_dev.c
create mode 100644 fs/lustre/obdclass/jobid.cSignature · 1 line
1.8.3.1
PATCH 1/151 lnet: fix needed headers for lnet headers
The linux kernel now has a sanity check for the global headers, activated by CONFIG_HEADER_TEST=y and CONFIG_KERNEL_HEADER_TEST=y. Some LNet headers are missing a few headers to allow them to build properly.
--- include/linux/libcfs/libcfs_fail.h | 1 + include/linux/libcfs/libcfs_private.h | 3 +++ include/uapi/linux/lnet/lnetst.h | 2 ++ 3 files changed, 6 insertions(+)
include/linux/libcfs/libcfs_fail.h +1 -0
@@ -35,6 +35,7 @@#include <linux/sched.h> #include <linux/wait.h> +#include <linux/libcfs/libcfs_debug.h> extern unsigned long cfs_fail_loc;extern unsigned int cfs_fail_val;
include/linux/libcfs/libcfs_private.h +3 -0
@@ -39,6 +39,9 @@#ifndef __LIBCFS_PRIVATE_H__#define __LIBCFS_PRIVATE_H__ +#include <linux/sched.h> +#include <linux/libcfs/libcfs_debug.h> +#ifndef DEBUG_SUBSYSTEM # define DEBUG_SUBSYSTEM S_UNDEFINED#endif
include/uapi/linux/lnet/lnetst.h +2 -0
@@ -38,6 +38,8 @@#define __LNET_ST_H__#include <linux/types.h> +#include <linux/time.h> +#include <linux/lnet/lnet-types.h> #define LST_FEAT_NONE (0)#define LST_FEAT_BULK_LEN (1 << 0)/* enable variable page size */
Signature · 1 line
1.8.3.1
PATCH 1/151 lnet: fix needed headers for lnet headers
Quoted 7 lines · James Simmons
The linux kernel now has a sanity check for the global headers, activated by
CONFIG_HEADER_TEST=yandCONFIG_KERNEL_HEADER_TEST=y. Some LNet headers are missing a few headers to allow them to build properly.Signed-off-by James Simmons <[email protected]>Signed-off-by NeilBrown <[email protected]>
I've added the following to this patch
include/linux/libcfs/libcfs_debug.h +2 -0
@@ -56,6 +56,8 @@ extern unsigned int libcfs_console_backoff;extern unsigned int libcfs_debug_binary;extern char libcfs_debug_file_path_arr[PATH_MAX ]; +struct task_struct ; +int libcfs_debug_mask2str (char *str,int size,int mask,int is_subsys);int libcfs_debug_str2mask (int *mask,const char *str,int is_subsys);void libcfs_debug_dumpstack (struct task_struct *tsk);
NeilBrown
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20191001/90609b23/attachment.sig>
PATCH 1/151 lnet: fix needed headers for lnet headers
Quoted 28 lines · James Simmons
Quoted 8 lines
The linux kernel now has a sanity check for the global headers, activated by
CONFIG_HEADER_TEST=yandCONFIG_KERNEL_HEADER_TEST=y. Some LNet headers are missing a few headers to allow them to build properly.Signed-off-by James Simmons <[email protected]>Signed-off-by NeilBrown <[email protected]>I've added the following to this patch
include/linux/libcfs/libcfs_debug.h +2
-0@@ -56,6 +56,8 @@ extern unsigned int libcfs_console_backoff;extern unsigned int libcfs_debug_binary;extern char libcfs_debug_file_path_arr[PATH_MAX ]; +struct task_struct ; +int libcfs_debug_mask2str (char *str,int size,int mask,int is_subsys);int libcfs_debug_str2mask (int *mask,const char *str,int is_subsys);void libcfs_debug_dumpstack (struct task_struct *tsk);
Quoted 1 line
NeilBrown
PATCH 2/151 lustre: fix signal handling in abortable waits.
In reply to James SimmonsFrom: NeilBrown <[email protected]>
Instead of blocking all but the fatal signals, we were blocking just the fatal signals. That wasn't what was intended, but is easy to fix.
--- fs/lustre/include/lustre_lib.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/lustre/include/lustre_lib.h +3 -3
@@ -93,7 +93,7 @@ static inline int l_fatal_signal_pending(struct task_struct *p) ({ \ sigset_t __new_blocked, __old_blocked; \ int __ret = 0; \ - siginitset(&__new_blocked, LUSTRE_FATAL_SIGS); \ + siginitsetinv(&__new_blocked, LUSTRE_FATAL_SIGS); \ sigprocmask(SIG_BLOCK, &__new_blocked, &__old_blocked); \ __ret = wait_event_interruptible(wq, condition); \ sigprocmask(SIG_SETMASK, &__old_blocked, NULL); \
@@ -104,7 +104,7 @@ static inline int l_fatal_signal_pending(struct task_struct *p) ({ \ sigset_t __new_blocked, __old_blocked; \ int __ret = 0; \ - siginitset(&__new_blocked, LUSTRE_FATAL_SIGS); \ + siginitsetinv(&__new_blocked, LUSTRE_FATAL_SIGS); \ sigprocmask(SIG_BLOCK, &__new_blocked, &__old_blocked); \ __ret = wait_event_interruptible_timeout(wq, condition, timeout);\ sigprocmask(SIG_SETMASK, &__old_blocked, NULL); \
@@ -115,7 +115,7 @@ static inline int l_fatal_signal_pending(struct task_struct *p) ({ \ sigset_t __new_blocked, __old_blocked; \ int __ret = 0; \ - siginitset(&__new_blocked, LUSTRE_FATAL_SIGS); \ + siginitsetinv(&__new_blocked, LUSTRE_FATAL_SIGS); \ sigprocmask(SIG_BLOCK, &__new_blocked, &__old_blocked); \ __ret = wait_event_interruptible_exclusive(wq, condition); \ sigprocmask(SIG_SETMASK, &__old_blocked, NULL); \
Signature · 1 line
1.8.3.1
PATCH 3/151 lnet: ksocklnd: add secondary IP address handling
In reply to James SimmonsWith the current ksocknal_enumerate_interfaces() only primary IP addresses are returned. This disables using network aliasing which some people use. Remove the secondary IP address filtering. Update the string handling since ifa_device names can be different than the net_device name. Discard the 'j' counter and instead keep ksnn_ninterfaces up to date. This means that we return 0 on success, rather than a count of added interfaces. Update the too many interfaces test with a better test using ARRAY_SIZE.
WC-bug-id: https://jira.whamcloud.com/browse/LU-11893
Lustre-commit: 9a2013af0668 ("LU-11893 ksocklnd: add secondary IP address handling
Reviewed-on: https://review.whamcloud.com/34392
--- net/lnet/klnds/socklnd/socklnd.c | 75 +++++++++++++++++++++++++++------------- 1 file changed, 51 insertions(+), 24 deletions(-)
net/lnet/klnds/socklnd/socklnd.c +51 -24
@@ -2604,54 +2604,58 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id)static int ksocknal_enumerate_interfaces (struct ksock_net *net,char *iname) { -int j =0 ;struct net_device *dev;rtnl_lock ();for_each_netdev (&init_net, dev) { -const char *name = dev->name; +/* The iname specified by a user land configuration can +* map to an ifa_label so always treat iname as an ifa_label. +* If iname is NULL then fall back to the net device name. +*/ +const char *name = iname ? iname : dev->name;const struct in_ifaddr *ifa;struct in_device *in_dev; -struct ksock_interface *ksi = - &net->ksnn_interfaces[net->ksnn_ninterfaces + j]; -if (strcmp (name,"lo" ) ==0 )/* skip the loopback IF */ -continue ; -if (iname &&strcmp (name, iname) !=0 ) +if (strcmp (dev->name,"lo" ) ==0 )/* skip the loopback IF */ continue ;if (!(dev_get_flags (dev) &IFF_UP )) { -CWARN ("Ignoring interface %s (down)\n" , name); +CWARN ("Ignoring interface %s (down)\n" , dev->name);continue ; } -if (j ==LNET_INTERFACES_NUM ) { -CWARN ("Ignoring interface %s (too many interfaces)\n" , - name); -continue ; - } in_dev =__in_dev_get_rtnl (dev);if (!in_dev) { -CWARN ("Interface %s has no IPv4 status.\n" , name); +CWARN ("Interface %s has no IPv4 status.\n" , dev->name);continue ; } -in_dev_for_each_ifa_rcu (ifa, in_dev) -if (!(ifa->ifa_flags &IFA_F_SECONDARY ) && -strcmp (ifa->ifa_label, name) ==0 ) { + +in_dev_for_each_ifa_rcu (ifa, in_dev) { +if (strcmp (name, ifa->ifa_label) ==0 ) { +int idx = net->ksnn_ninterfaces; +struct ksock_interface *ksi; + +if (idx >=ARRAY_SIZE (net->ksnn_interfaces)) { +rtnl_unlock (); +return -E2BIG ; + } + + ksi = &net->ksnn_interfaces[idx]; ksi->ksni_ipaddr =ntohl (ifa->ifa_local); ksi->ksni_netmask = ifa->ifa_mask;strlcpy (ksi->ksni_name, name,sizeof (ksi->ksni_name)); - j++; + net->ksnn_ninterfaces++;break ; } + } }rtnl_unlock (); -if (!iname && !j) +if (net->ksnn_ninterfaces ==0 )CERROR ("Can't find any usable interfaces\n" ); -return j; +return net->ksnn_ninterfaces >0 ?0 : -ENOENT ; } static int
@@ -2810,17 +2814,40 @@ static int ksocknal_push(struct lnet_ni *ni, struct lnet_process_id id) rc =ksocknal_enumerate_interfaces (net,NULL );if (rc <=0 ) goto fail_1; - - net->ksnn_ninterfaces = rc; }else { +/* Before Multi-Rail ksocklnd would manage +* multiple interfaces with its own tcp bonding. +* If we encounter an old configuration using +* this tcp bonding approach then we need to +* handle more than one ni_interfaces. +* +* In Multi-Rail configuration only ONE ni_interface +* should exist. Each IP alias should be mapped to +* each 'struct net_ni'. +*/ for (i =0 ; i <LNET_INTERFACES_NUM ; i++) { +int j; +if (!ni->ni_interfaces[i])break ; - rc =ksocknal_enumerate_interfaces (net, ni->ni_interfaces[i]); +for (j =0 ; j < net->ksnn_ninterfaces; j++) { +struct ksock_interface *ksi; + + ksi = &net->ksnn_interfaces[j]; + +if (strcmp (ni->ni_interfaces[i], + ksi->ksni_name) ==0 ) { +CERROR ("found duplicate %s\n" , + ksi->ksni_name); + rc = -EEXIST ; + goto fail_1; + } + } + + rc =ksocknal_enumerate_interfaces (net, ni->ni_interfaces[i]);if (rc <=0 ) goto fail_1; - net->ksnn_ninterfaces += rc; } }
Signature · 1 line
1.8.3.1
PATCH 4/151 lnet: o2iblnd: add secondary IP address handling
In reply to James SimmonsUsing dev_get_by_name() in kiblnd_create_dev() means we can only discover primary IP addresses. This breaks using network aliasing which some people use. Move away from dev_get_by_name() to using for_ifa() so we can detect any secondary IP addresses.
WC-bug-id: https://jira.whamcloud.com/browse/LU-11893
Linux-commit: c4b39bf56bbc ("LU-11893 o2iblnd: add secondary IP address handling")
Reviewed-on: https://review.whamcloud.com/34476
WC-bug-id: https://jira.whamcloud.com/browse/LU-12381
Linux-commit: 1dea5aac9d9b ("LU-11893 o2iblnd: ignore down interfaces")
Reviewed-on: https://review.whamcloud.com/35098
--- net/lnet/klnds/o2iblnd/o2iblnd.c | 78 ++++++++++++++++++++++------------------ 1 file changed, 44 insertions(+), 34 deletions(-)
net/lnet/klnds/o2iblnd/o2iblnd.c +44 -34
@@ -2557,53 +2557,58 @@ static struct kib_dev *kiblnd_create_dev(char *ifname) {const struct in_ifaddr *ifa;struct net_device *netdev; -struct in_device *in_dev; -struct kib_dev *dev; +struct kib_dev *dev =NULL ;int flags;int rc;rtnl_lock (); - netdev =dev_get_by_name (&init_net, ifname); -if (!netdev) { -CERROR ("Can't find IPoIB interface %s\n" , - ifname); - goto unlock; - } +for_each_netdev (&init_net, netdev) { +struct in_device *in_dev; - flags =dev_get_flags (netdev); -if (!(flags &IFF_UP )) { -CERROR ("Can't query IPoIB interface %s: it's down\n" , ifname); - goto unlock; - } +if (strcmp (netdev->name,"lo" ) ==0 )/* skip the loopback IF */ +continue ; - dev =kzalloc (sizeof (*dev),GFP_NOFS ); -if (!dev) - goto unlock; + flags =dev_get_flags (netdev); +if (!(flags &IFF_UP )) { +CWARN ("Can't query IPoIB interface %s: it's down\n" , + netdev->name); +continue ; + } - dev->ibd_can_failover = !!(flags &IFF_MASTER ); + in_dev =__in_dev_get_rtnl (netdev); +if (!in_dev) { +CWARN ("Interface %s has no IPv4 status.\n" , + netdev->name); +continue ; + } -INIT_LIST_HEAD (&dev->ibd_nets); -INIT_LIST_HEAD (&dev->ibd_list);/* not yet in kib_devs */ -INIT_LIST_HEAD (&dev->ibd_fail_list); +in_dev_for_each_ifa_rcu (ifa, in_dev) { +if (strcmp (ifname, ifa->ifa_label) ==0 ) { + dev =kzalloc (sizeof (*dev),GFP_NOFS ); +if (!dev) + goto unlock; - in_dev =__in_dev_get_rtnl (netdev); -if (!in_dev) { -kfree (dev); - goto unlock; - } + dev->ibd_can_failover = !!(flags &IFF_MASTER ); + dev->ibd_ifip =ntohl (ifa->ifa_local); -in_dev_for_each_ifa_rcu (ifa, in_dev) -if (!(ifa->ifa_flags &IFA_F_SECONDARY ) && -strcmp (ifa->ifa_label, ifname) ==0 ) { - dev->ibd_ifip =ntohl (ifa->ifa_local); -break ; +INIT_LIST_HEAD (&dev->ibd_nets); +/* not yet in kib_devs */ +INIT_LIST_HEAD (&dev->ibd_list); +INIT_LIST_HEAD (&dev->ibd_fail_list); +break ; + } } + }rtnl_unlock (); +if (!dev) { +CERROR ("Can't find any usable interfaces\n" ); +return NULL ; + } +if (dev->ibd_ifip ==0 ) {CERROR ("Can't initialize device: no IP address\n" ); -kfree (dev); -return NULL ; + goto free_dev; }strcpy (&dev->ibd_ifname[0 ], ifname);
@@ -2611,14 +2616,15 @@ static struct kib_dev *kiblnd_create_dev(char *ifname) rc =kiblnd_dev_failover (dev);if (rc) {CERROR ("Can't initialize device: %d\n" , rc); -kfree (dev); -return NULL ; + goto free_dev; }list_add_tail (&dev->ibd_list, &kiblnd_data.kib_devs);return dev; unlock:rtnl_unlock (); +free_dev: +kfree (dev);return NULL ; }
@@ -2968,6 +2974,10 @@ static int kiblnd_startup(struct lnet_ni *ni)if (rc) goto net_failed; +/* ni_interfaces is only to support legacy pre Multi-Rail +* tcp bonding for ksocklnd. Multi-Rail wants each secondary +* IP to be treated as an unique 'struct ni' interfaces instead. +*/ if (ni->ni_interfaces[0 ]) {/* Use the IPoIB interface specified in 'networks=' */
Signature · 1 line
1.8.3.1
PATCH 5/151 lnet: consoldate secondary IP address handling
In reply to James SimmonsThe last piece of code with broken secondary IP address support is lnet_parse_ip2nets(). We could fix it like o2iblnd or socklnd was done but since the LND drivers resolved those issues instead we can move the handling out of the LND drivers into one place in the LNet core. To do this we introduce struct lnet_inetdev which is a collection of data that the current LNet layer requires. The new function lnet_inet_enumerate() is used to collect this information.
WC-bug-id: https://jira.whamcloud.com/browse/LU-11893
Linux-commit: b770d7117f35a ("LU-11893 lnet: consoldate secondary IP address handling")
Reviewed-on: https://review.whamcloud.com/34993
--- include/linux/lnet/lib-lnet.h | 10 +++ net/lnet/klnds/o2iblnd/o2iblnd.c | 167 ++++++++++----------------------------- net/lnet/klnds/socklnd/socklnd.c | 121 +++++++++++----------------- net/lnet/lnet/config.c | 121 ++++++++++++++++------------ 4 files changed, 170 insertions(+), 249 deletions(-)
include/linux/lnet/lib-lnet.h +10 -0
@@ -36,6 +36,7 @@#ifndef __LNET_LIB_LNET_H__#define __LNET_LIB_LNET_H__ +#include <linux/netdevice.h> #include <linux/libcfs/libcfs.h> #include <linux/libcfs/libcfs_cpu.h> #include <linux/libcfs/libcfs_string.h>
@@ -649,6 +650,15 @@ void lnet_connect_console_error(int rc, lnet_nid_t peer_nid,int lnet_acceptor_start (void );void lnet_acceptor_stop (void ); +struct lnet_inetdev { +u32 li_cpt; +u32 li_flags; +u32 li_ipaddr; +u32 li_netmask; +char li_name[IFNAMSIZ ]; +}; + +int lnet_inet_enumerate (struct lnet_inetdev **dev_list);int lnet_sock_setbuf (struct socket *socket,int txbufsize,int rxbufsize);int lnet_sock_getbuf (struct socket *socket,int *txbufsize,int *rxbufsize);int lnet_sock_getaddr (struct socket *socket,bool remote,u32 *ip,int *port);
net/lnet/klnds/o2iblnd/o2iblnd.c +43 -124
225 lines not on this page · raw message
net/lnet/klnds/socklnd/socklnd.c +45 -76
158 lines not on this page · raw message
net/lnet/lnet/config.c +72 -49
169 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 6/151 lustre: support for gcc8
In reply to James SimmonsFrom: Alex Zhuravlev <[email protected]>
Building with gcc8 reports new warnings which are addressed in this patch.
WC-bug-id: https://jira.whamcloud.com/browse/LU-11233
Lustre-commit: 6601661f9632 ("LU-11233 build: support for gcc8")
Reviewed-on: https://review.whamcloud.com/34660
--- fs/lustre/lmv/lmv_obd.c | 14 ++++++++------ fs/lustre/obdclass/genops.c | 2 +- net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 3 +-- 3 files changed, 10 insertions(+), 9 deletions(-)
fs/lustre/lmv/lmv_obd.c +8 -6
@@ -640,6 +640,7 @@ static int lmv_fid2path(struct obd_export *exp, int len, void *karg,if (remote_gf) {struct getinfo_fid2path *ori_gf;char *ptr; +int len; ori_gf = karg; if (strlen(ori_gf->gf_path) + 1 +
@@ -650,12 +651,13 @@ static int lmv_fid2path(struct obd_export *exp, int len, void *karg, ptr = ori_gf->gf_path; -memmove (ptr +strlen (gf->gf_path) +1 , ptr, -strlen (ori_gf->gf_path)); - -strncpy (ptr, gf->gf_path,strlen (gf->gf_path)); - ptr +=strlen (gf->gf_path); - *ptr ='/' ; + len =strlen (gf->gf_path); +/* move the current path to the right to release space +* for closer-to-root part +*/ +memmove (ptr + len +1 , ptr,strlen (ori_gf->gf_path)); +memcpy (ptr, gf->gf_path, len); + ptr[len] ='/' ; } CDEBUG(D_INFO, "%s: get path %s " DFID " rec: %llu ln: %u\n",
fs/lustre/obdclass/genops.c +1 -1
@@ -322,7 +322,7 @@ struct obd_device *class_newdev(const char *type_name, const char *name, newdev->obd_conn_inprogress =0 ; -strncpy (newdev->obd_uuid.uuid, uuid,strlen (uuid)); +strncpy (newdev->obd_uuid.uuid, uuid,UUID_MAX );CDEBUG (D_IOCTL ,"Allocate new device %s (%p)\n" , newdev->obd_name, newdev);
net/lnet/klnds/o2iblnd/o2iblnd_cb.c +1 -2
@@ -1763,7 +1763,7 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,struct kib_tx *tx;int nob;int post_credit =IBLND_POSTRX_PEER_CREDIT ; -u64 ibprm_cookie; +u64 ibprm_cookie = rxmsg->ibm_u.putreq.ibprm_cookie;int rc =0 ;LASSERT (iov_iter_count (to) <= rlen);
@@ -1796,7 +1796,6 @@ static int kiblnd_resolve_addr(struct rdma_cm_id *cmid,break ;case IBLND_MSG_PUT_REQ : { -u64 ibprm_cookie = rxmsg->ibm_u.putreq.ibprm_cookie;struct kib_msg *txmsg;struct kib_rdma_desc *rd;
Signature · 1 line
1.8.3.1
PATCH 7/151 lnet: Allocate MEs and small MDs in own kmem_caches
In reply to James SimmonsFrom: Bruno Faccini <[email protected]>
As part of LU-3848 and LU-4330, it has been discovered that LNET MEs and small MDs (<=128 Bytes) are allocated in <size-128> kmem_cache and thus can suffer quite frequent corruptions, from other modules or Kernel parts, that occur there. To avoid this, MEs and small-MDs specific kmem_cache have been created.
Also, lnet_libmd struct fields have been re-ordered to optimize its memory foot-print.
WC-bug-id: https://jira.whamcloud.com/browse/LU-4330
Lustre-commit: 9d9bb678d6b3 ("LU-4330 lnet: Allocate MEs and small MDs in own kmem_caches")
Reviewed-on: http://review.whamcloud.com/18586
--- include/linux/lnet/lib-lnet.h | 39 ++++++++++++++++++++++++++++++++++++++- include/linux/lnet/lib-types.h | 2 +- net/lnet/lnet/api-ni.c | 40 ++++++++++++++++++++++++++++++++++++++++ net/lnet/lnet/lib-md.c | 2 +- net/lnet/lnet/lib-me.c | 2 +- net/lnet/lnet/lib-ptl.c | 2 +- 6 files changed, 82 insertions(+), 5 deletions(-)
include/linux/lnet/lib-lnet.h +38 -1
@@ -80,6 +80,12 @@/* default timeout */ #define DEFAULT_PEER_TIMEOUT 180 +#define LNET_SMALL_MD_SIZE offsetof(struct lnet_libmd, md_iov.iov[1]) +extern struct kmem_cache *lnet_mes_cachep;/* MEs kmem_cache */ +extern struct kmem_cache *lnet_small_mds_cachep;/* <= LNET_SMALL_MD_SIZE bytes +* MDs kmem_cache +*/ +static inline int lnet_is_route_alive (struct lnet_route *route) {/* gateway is down */
@@ -151,6 +157,24 @@ static inline int lnet_md_unlinkable(struct lnet_libmd *md)cfs_percpt_unlock (the_lnet.ln_res_lock, cpt); } +static inline void +lnet_md_free (struct lnet_libmd *md) +{ +unsigned int size; + +if ((md->md_options &LNET_MD_KIOV ) !=0 ) + size = offsetof(struct lnet_libmd, md_iov.kiov[md->md_niov]); +else + size = offsetof(struct lnet_libmd, md_iov.iov[md->md_niov]); + +if (size <=LNET_SMALL_MD_SIZE ) { +CDEBUG (D_MALLOC ,"slab-freed 'md' at %p.\n" , md); +kmem_cache_free (lnet_small_mds_cachep, md); + }else { +kfree (md); + } +} +static inline int lnet_res_lock_current (void ) {
@@ -205,7 +229,20 @@ static inline int lnet_md_unlinkable(struct lnet_libmd *md) size = offsetof(struct lnet_libmd, md_iov.iov[niov]); } - md =kzalloc (size,GFP_NOFS ); +if (size <=LNET_SMALL_MD_SIZE ) { + md =kmem_cache_alloc (lnet_small_mds_cachep, +GFP_NOFS | __GFP_ZERO); +if (md) { +CDEBUG (D_MALLOC , +"slab-alloced 'md' of size %u@%p.\n" , + size, md); + }else { +CDEBUG (D_MALLOC ,"failed to allocate 'md' of size %u\n" , + size); + } + }else { + md =kzalloc (size,GFP_NOFS ); + }if (md) {/* Set here in case of early free */ md->md_options = umd->options;
include/linux/lnet/lib-types.h +1 -1
@@ -164,9 +164,9 @@ struct lnet_libmd {int md_refcount;unsigned int md_options;unsigned int md_flags; +unsigned int md_niov;/* # frags at end of struct */ void *md_user_ptr;struct lnet_eq *md_eq; -unsigned int md_niov;/* # frags */ struct lnet_handle_md md_bulk_handle;union {struct kvec iov[LNET_MAX_IOV ];
net/lnet/lnet/api-ni.c +40 -0
@@ -216,6 +216,41 @@ static int lnet_discover(struct lnet_process_id id, u32 force,mutex_init (&the_lnet.ln_lnd_mutex); } +struct kmem_cache *lnet_mes_cachep;/* MEs kmem_cache */ +struct kmem_cache *lnet_small_mds_cachep;/* <= LNET_SMALL_MD_SIZE bytes +* MDs kmem_cache +*/ + +static int +lnet_descriptor_setup (void ) +{ +/* create specific kmem_cache for MEs and small MDs (i.e., originally +* allocated in <size-xxx> kmem_cache). +*/ + lnet_mes_cachep =kmem_cache_create ("lnet_MEs" ,sizeof (struct lnet_me ), +0 ,0 ,NULL ); +if (!lnet_mes_cachep) +return -ENOMEM ; + + lnet_small_mds_cachep =kmem_cache_create ("lnet_small_MDs" , +LNET_SMALL_MD_SIZE ,0 ,0 , +NULL ); +if (!lnet_small_mds_cachep) +return -ENOMEM ; + +return 0 ; +} + +static void +lnet_descriptor_cleanup (void ) +{ +kmem_cache_destroy (lnet_small_mds_cachep); + lnet_small_mds_cachep =NULL ; + +kmem_cache_destroy (lnet_mes_cachep); + lnet_mes_cachep =NULL ; +} +static int lnet_create_remote_nets_table (void ) {
@@ -701,6 +736,10 @@ struct lnet_libhandle *INIT_LIST_HEAD (&the_lnet.ln_dc_expired);init_waitqueue_head (&the_lnet.ln_dc_waitq); + rc =lnet_descriptor_setup (); +if (rc !=0 ) + goto failed; + rc =lnet_create_remote_nets_table ();if (rc) goto failed;
@@ -798,6 +837,7 @@ struct lnet_libhandle * the_lnet.ln_counters =NULL ; }lnet_destroy_remote_nets_table (); +lnet_descriptor_cleanup ();return 0 ; }
net/lnet/lnet/lib-md.c +1 -1
@@ -82,7 +82,7 @@LASSERT (!list_empty (&md->md_list));list_del_init (&md->md_list); -kfree (md); +lnet_md_free (md); }struct page *lnet_kvaddr_to_page (unsigned long vaddr)
net/lnet/lnet/lib-me.c +1 -1
@@ -91,7 +91,7 @@if (!mtable)/* can't match portal type */ return -EPERM ; - me =kzalloc (sizeof (*me),GFP_NOFS ); + me =kmem_cache_alloc (lnet_mes_cachep,GFP_NOFS | __GFP_ZERO);if (!me)return -ENOMEM ;
net/lnet/lnet/lib-ptl.c +1 -1
@@ -773,7 +773,7 @@ struct list_head * != NULL) {CERROR ("Active ME %p on exit\n" , me);list_del (&me->me_list); -kfree (me); +kmem_cache_free (lnet_mes_cachep, me); } }/* the extra entry is for MEs with ignore bits */
Signature · 1 line
1.8.3.1
PATCH 8/151 lustre: seq: make seq_proc_write_common() safer
In reply to James SimmonsFrom: Andreas Dilger <[email protected]>
Don't allow seq_proc_write_common() to specify arbitrary ranges, since this can permanently corrupt the sequence controller and/or sequnece server. That would allow duplicate FID allocation, or possibly prevent any new files to be created or servers to be added to the filesystem.
Instead, limit the sequence range that can be written via /proc to a subset of the sequence range currently allocated to that node. Add the "clear" keyword to allow dropping the entire local sequence and force a new one to be fetched from the sequence server.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3642
Lustre-commit: 05f69f5ee20e ("LU-3642 seq: make seq_proc_write_common() safer")
Reviewed-on: http://review.whamcloud.com/7123
--- fs/lustre/fid/lproc_fid.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
fs/lustre/fid/lproc_fid.c +7 -5
@@ -52,14 +52,18 @@/* Format: [0x64BIT_INT - 0x64BIT_INT] + 32 bytes just in case */ #define MAX_FID_RANGE_STRLEN (32 + 2 * 2 * sizeof(u64))/* -* Note: this function is only used for testing, it is no safe for production -* use. +* Reduce the SEQ range allocated to a node to a strict subset of the range +* currently-allocated SEQ range. If the specified range is "clear", then +* drop all allocated sequences and request a new one from the master. +* +* Note: this function should only be used for testing, it is not necessarily +* safe for production use. */ static int ldebugfs_fid_write_common (const char __user *buffer,size_t count,struct lu_seq_range *range) { -struct lu_seq_range tmp; +struct lu_seq_range tmp = {0 , };int rc;char kernbuf[MAX_FID_RANGE_STRLEN ];
@@ -82,8 +86,6 @@ rc =sscanf (kernbuf,"[%llx - %llx]\n" , (unsigned long long *)&tmp.lsr_start, (unsigned long long *)&tmp.lsr_end); -if (rc !=2 ) -return -EINVAL ;if (!lu_seq_range_is_sane (&tmp) ||lu_seq_range_is_zero (&tmp) || tmp.lsr_start < range->lsr_start || tmp.lsr_end > range->lsr_end)return -EINVAL ;
Signature · 1 line
1.8.3.1
PATCH 9/151 lustre: ptlrpc: Fix an rq_no_reply assertion failure
In reply to James SimmonsFrom: Li Wei <[email protected]>
An OSS had an assertion failure:
LustreError: 5366:0:(ldlm_lib.c:2689:target_bulk_io()) @@@ timeout on bulk GET after 0+0s req at ffff88083a61b400 x1476486691018500/t0(4300509964) o4->8dda3382-83f8-6445-5eea-828fd59e4a06 at 192.168.1.116@o2ib1:0/0 lens 504/448 e 391470 to 0 dl 1408494729 ref 2 fl Complete:/4/0 rc 0/0
LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) ASSERTION( req->rq_no_reply == 0 ) failed:
Lustre: soaked-OST0000: Bulk IO write error with 8dda3382-83f8-6445-5eea-828fd59e4a06 (at 192.168.1.116 at o2ib1), client will retry: rc -110
LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) LBUG
Pid: 5432, comm: ll_ost_io03_003
Call Trace: [<ffffffffa0641895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [<ffffffffa0641e97>] lbug_with_loc+0x47/0xb0 [libcfs] [<ffffffffa09cda4c>] ptlrpc_send_reply+0x4ec/0x7f0 [ptlrpc] [<ffffffffa09d4aae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [<ffffffffa09e4d75>] ptlrpc_at_check_timed+0xcd5/0x1370 [ptlrpc] [<ffffffffa09dc1e9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] [<ffffffffa09e66f8>] ptlrpc_main+0x12e8/0x1990 [ptlrpc] [<ffffffff81069290>] ? pick_next_task_fair+0xd0/0x130 [<ffffffff81529246>] ? schedule+0x176/0x3b0 [<ffffffffa09e5410>] ? ptlrpc_main+0x0/0x1990 [ptlrpc] [<ffffffff8109abf6>] kthread+0x96/0xa0 [<ffffffff8100c20a>] child_rip+0xa/0x20 [<ffffffff8109ab60>] ? kthread+0x0/0xa0 [<ffffffff8100c200>] ? child_rip+0x0/0x20
The thread in tgt_brw_write() had decided not to reply by setting rq_no_reply, right before another thread tried to send an early reply for the request.
WC-bug-id: https://jira.whamcloud.com/browse/LU-5537
Lustre-commit: a8d448e4cd59 ("LU-5537 ptlrpc: Fix an rq_no_reply assertion failure")
Reviewed-on: http://review.whamcloud.com/11740
--- fs/lustre/ptlrpc/service.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
fs/lustre/ptlrpc/service.c +10 -0
@@ -1098,6 +1098,16 @@ static int ptlrpc_at_send_early_reply(struct ptlrpc_request *req) reqcopy->rq_reqmsg = reqmsg;memcpy (reqmsg, req->rq_reqmsg, req->rq_reqlen); +/* +* tgt_brw_read() and tgt_brw_write() may have decided not to reply. +* Without this check, we would fail the rq_no_reply assertion in +* ptlrpc_send_reply(). +*/ +if (reqcopy->rq_no_reply) { + rc = -ETIMEDOUT ; + goto out; + } +LASSERT (atomic_read (&req->rq_refcount));/** if it is last refcount then early reply isn't needed */ if (atomic_read (&req->rq_refcount) ==1 ) {
Signature · 1 line
1.8.3.1
PATCH 10/151 lustre: fld: resend seq lookup RPC if it is on LWP
In reply to James SimmonsFrom: Wang Di <[email protected]>
Because Light Weight connection might be evicted after restart, then cause inflight RPC fails, to avoid this, we need resend seq lookup RPC.
remove "-f" from "stop mdt" in sanity 17m, so umount can keep the the connection, and otherwise the OSP might be evicted.
WC-bug-id: https://jira.whamcloud.com/browse/LU-4571
Lustre-commit: cf7f66d87e52 ("LU-4571 fld: resend seq lookup RPC if it is on LWP")
Reviewed-on: http://review.whamcloud.com/9106
--- fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
fs/lustre/fld/fld_request.c +21 -2
@@ -314,6 +314,7 @@ int fld_client_rpc(struct obd_export *exp,LASSERT (exp); +again: imp =class_exp2cliimp (exp);switch (fld_op) {case FLD_QUERY :
@@ -329,8 +330,15 @@ int fld_client_rpc(struct obd_export *exp, op =req_capsule_client_get (&req->rq_pill, &RMF_FLD_OPC ); *op =FLD_LOOKUP ; -if (imp->imp_connect_flags_orig &OBD_CONNECT_MDS_MDS ) +/* For MDS_MDS seq lookup, it will always use LWP connection, +* but LWP will be evicted after restart, so cause the error. +* so we will set no_delay for seq lookup request, once the +* request fails because of the eviction. always retry here +*/ +if (imp->imp_connect_flags_orig &OBD_CONNECT_MDS_MDS ) { req->rq_allow_replay =1 ; + req->rq_no_delay =1 ; + }break ;case FLD_READ : req = ptlrpc_request_alloc_pack(imp, &RQF_FLD_READ,
@@ -358,8 +366,19 @@ int fld_client_rpc(struct obd_export *exp,obd_get_request_slot (&exp->exp_obd->u.cli); rc =ptlrpc_queue_wait (req);obd_put_request_slot (&exp->exp_obd->u.cli); -if (rc) +if (rc !=0 ) { +if (rc == -EWOULDBLOCK ) { +/* For no_delay req(see above), EWOULDBLOCK means the +* connection is being evicted, but this seq lookup +* should not return error, since it would cause +* unnecessary failure of the application, instead +* it should retry here +*/ +ptlrpc_req_finished (req); + goto again; + } goto out_req; + }if (fld_op ==FLD_QUERY ) { prange =req_capsule_server_get (&req->rq_pill, &RMF_FLD_MDFLD );
Signature · 1 line
1.8.3.1
PATCH 11/151 lustre: fld: retry fld rpc even for ESHUTDOWN
In reply to James SimmonsFrom: Wang Di <[email protected]>
when LWP is being evicted, because it is not replayable, the request might return ESHUTDOWN or EWOULDBLOCK, instead of failed, which might cause application failure, fld client will retry RPC, until the connection is being setup again or the LWP is being closed.
WC-bug-id: https://jira.whamcloud.com/browse/LU-4420
Lustre-commit: d335e310d4bf ("LU-4420 fld: retry fld rpc even for ESHUTDOWN")
Reviewed-on: http://review.whamcloud.com/10285
--- fs/lustre/fld/fld_request.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
fs/lustre/fld/fld_request.c +6 -6
@@ -367,12 +367,12 @@ int fld_client_rpc(struct obd_export *exp, rc =ptlrpc_queue_wait (req);obd_put_request_slot (&exp->exp_obd->u.cli);if (rc !=0 ) { -if (rc == -EWOULDBLOCK ) { -/* For no_delay req(see above), EWOULDBLOCK means the -* connection is being evicted, but this seq lookup -* should not return error, since it would cause -* unnecessary failure of the application, instead -* it should retry here +if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN ) { +/* For no_delay req(see above), EWOULDBLOCK and +* ESHUTDOWN means the connection is being evicted, +* but this seq lookup should not return error, +* since it would cause unnecessary failure of the +* application, instead it should retry here */ ptlrpc_req_finished (req); goto again;
Signature · 1 line
1.8.3.1
PATCH 12/151 lustre: fld: retry fld rpc until the import is closed
In reply to James SimmonsFrom: Wang Di <[email protected]>
Since LWP is not replayable, it should retry RPC until the import is closed, otherwise it would cause unnecessary failure of the application.
WC-bug-id: https://jira.whamcloud.com/browse/LU-5312
Lustre-commit: 07d481957c16 ("LU-5312 fld: retry fld rpc until the import is closed")
Reviewed-on: http://review.whamcloud.com/11039
--- fs/lustre/fld/fld_request.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
fs/lustre/fld/fld_request.c +4 -6
@@ -367,12 +367,10 @@ int fld_client_rpc(struct obd_export *exp, rc =ptlrpc_queue_wait (req);obd_put_request_slot (&exp->exp_obd->u.cli);if (rc !=0 ) { -if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN ) { -/* For no_delay req(see above), EWOULDBLOCK and -* ESHUTDOWN means the connection is being evicted, -* but this seq lookup should not return error, -* since it would cause unnecessary failure of the -* application, instead it should retry here +if (imp->imp_state !=LUSTRE_IMP_CLOSED ) { +/* Since LWP is not replayable, so it will keep +* trying unless umount happens, otherwise it would +* cause unnecessary failure of the application. */ ptlrpc_req_finished (req); goto again;
Signature · 1 line
1.8.3.1
PATCH 13/151 lustre: fld: fld client lookup should retry
In reply to James SimmonsFrom: Wang Di <[email protected]>
If FLD client lookup fails because of the remote target is shutdown (or deactive), it should retry another target, otherwise it will cause the application failure.
And FLD client should stop retry if the import has been deactive.
WC-bug-id: https://jira.whamcloud.com/browse/LU-6419
Lustre-commit: 3ededde903c ("LU-6419 fld: fld client lookup should retry")
Reviewed-on: http://review.whamcloud.com/14313
--- fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
fs/lustre/fld/fld_request.c +21 -2
@@ -367,7 +367,7 @@ int fld_client_rpc(struct obd_export *exp, rc =ptlrpc_queue_wait (req);obd_put_request_slot (&exp->exp_obd->u.cli);if (rc !=0 ) { -if (imp->imp_state !=LUSTRE_IMP_CLOSED ) { +if (imp->imp_state !=LUSTRE_IMP_CLOSED && !imp->imp_deactive) {/* Since LWP is not replayable, so it will keep * trying unless umount happens, otherwise it would * cause unnecessary failure of the application.
@@ -404,6 +404,7 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds, {struct lu_seq_range res = {0 };struct lu_fld_target *target; +struct lu_fld_target *origin;int rc; rc =fld_cache_lookup (fld->lcf_cache, seq, &res);
@@ -415,7 +416,8 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,/* Can not find it in the cache */ target =fld_client_get_target (fld, seq);LASSERT (target); - + origin = target; +again:CDEBUG (D_INFO ,"%s: Lookup fld entry (seq: %#llx) on target %s (idx %llu)\n" , fld->lcf_name, seq,fld_target_name (target), target->ft_idx);
@@ -424,6 +426,23 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,fld_range_set_type (&res, flags); rc =fld_client_rpc (target->ft_exp, &res,FLD_QUERY ,NULL ); +if (rc == -ESHUTDOWN ) { +/* If fld lookup failed because the target has been shutdown, +* then try next target in the list, until trying all targets +* or fld lookup succeeds +*/ +spin_lock (&fld->lcf_lock); +if (target->ft_chain.next == fld->lcf_targets.prev) + target =list_entry (fld->lcf_targets.next, + struct lu_fld_target, ft_chain); +else + target =list_entry (target->ft_chain.next, + struct lu_fld_target, + ft_chain); +spin_unlock (&fld->lcf_lock); +if (target != origin) + goto again; + }if (rc ==0 ) { *mds = res.lsr_index;
Signature · 1 line
1.8.3.1
PATCH 14/151 lustre: ldlm: testcases for multiple modify RPCs feature
In reply to James SimmonsFrom: Gregoire Pichon <[email protected]>
This patch creates new testcases in the Auster test suite to test the support of multiple modify RPCs in flight feature.
Two new OBD_FAIL codes are added to allow several failure occurences of the reint requests or replies. This is needed because the current fail checks impose the OBD_FAIL_ONCE flag.
WC-bug-id: https://jira.whamcloud.com/browse/LU-5319
Lustre-commit: c2d27a0f1268 ("LU-5319 tests: testcases for multiple modify RPCs feature")
Reviewed-on: http://review.whamcloud.com/14861
--- fs/lustre/include/obd_support.h | 2 ++ fs/lustre/ldlm/ldlm_lib.c | 3 +++ 2 files changed, 5 insertions(+)
fs/lustre/include/obd_support.h +2 -0
@@ -189,6 +189,8 @@#define OBD_FAIL_MDS_SWAP_LAYOUTS_NET 0x14f#define OBD_FAIL_MDS_HSM_ACTION_NET 0x150#define OBD_FAIL_MDS_CHANGELOG_INIT 0x151 +#define OBD_FAIL_MDS_REINT_MULTI_NET 0x159 +#define OBD_FAIL_MDS_REINT_MULTI_NET_REP 0x15a/* layout lock */ #define OBD_FAIL_MDS_NO_LL_GETATTR 0x170
fs/lustre/ldlm/ldlm_lib.c +3 -0
@@ -689,6 +689,9 @@ int target_pack_pool_reply(struct ptlrpc_request *req)DEBUG_REQ (D_ERROR , req,"dropping reply" );return -ECOMM ; } +if (unlikely (lustre_msg_get_opc (req->rq_reqmsg) ==MDS_REINT && +OBD_FAIL_CHECK (OBD_FAIL_MDS_REINT_MULTI_NET_REP ))) +return -ECOMM ;if (unlikely (rc)) {DEBUG_REQ (D_NET , req,"processing error (%d)" , rc);
Signature · 1 line
1.8.3.1
PATCH 15/151 lustre: ldlm: Don't check opcode with NULL rq_reqmsg
In reply to James SimmonsFrom: Jeremy Filizetti <[email protected]>
When GSS is enabled it's possible to have a NULL rq_reqmsg if a bad signature or no context is returned during the unwrap of the request. Don't check the opcode in this case.
WC-bug-id: https://jira.whamcloud.com/browse/LU-7508
Lustre-commit: 3f4572caef5f ("LU-7508 ldlm: Don't check opcode with NULL rq_reqmsg")
Reviewed-on: http://review.whamcloud.com/17414
--- fs/lustre/ldlm/ldlm_lib.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
fs/lustre/ldlm/ldlm_lib.c +6 -2
@@ -689,8 +689,12 @@ int target_pack_pool_reply(struct ptlrpc_request *req)DEBUG_REQ (D_ERROR , req,"dropping reply" );return -ECOMM ; } -if (unlikely (lustre_msg_get_opc (req->rq_reqmsg) ==MDS_REINT && -OBD_FAIL_CHECK (OBD_FAIL_MDS_REINT_MULTI_NET_REP ))) +/* We can have a null rq_reqmsg in the event of bad signature or +* no context when unwrapping +*/ +if (req->rq_reqmsg && +unlikely (lustre_msg_get_opc (req->rq_reqmsg) ==MDS_REINT && +OBD_FAIL_CHECK (OBD_FAIL_MDS_REINT_MULTI_NET_REP )))return -ECOMM ;if (unlikely (rc)) {
Signature · 1 line
1.8.3.1
PATCH 16/151 lustre: all: remove all Sun license and URL references
In reply to James SimmonsThe link to the GPL license at Sun is no longer there. Also Sun Inc is gone so lets remove all references to it in our source code.
WC-bug-id: https://jira.whamcloud.com/browse/LU-8648
Lustre-commit: 72057a3af19e ("LU-8648 all: remove all Sun license and URL references")
Reviewed-on: http://review.whamcloud.com/22800
--- fs/lustre/include/lustre_lmv.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/lustre/include/lustre_lmv.h +1 -1
@@ -16,7 +16,7 @@* * You should have received a copy of the GNU General Public License * version 2 along with this program; If not, see -* http://www.sun.com/software/products/lustre/docs/GPLv2.pdf +* http://www.gnu.org/licenses/gpl-2.0.html * * GPL HEADER END */
Signature · 1 line
1.8.3.1
PATCH 17/151 lustre: ldlm: Use interval tree to update kms
In reply to James SimmonsFrom: Patrick Farrell <[email protected]>
Currently, ldlm_extent_shift_kms does a linear search of the list of granted locks on the resource when looking for the lock to use to update the kms.
This can be avoided by using the interval trees which store the extents of granted locks.
We change the interval tree to store locks in reverse order, with lowest end first, and highest end last.
Then we can easily find the last lock-end that doesn't ignore kms, and use that to choose a new kms.
NeilBrown: modified from original to work with linux interval-tree
WC-bug-id: https://jira.whamcloud.com/browse/LU-8272
Lustre-commit: e7cf1b060ba3 ("LU-8272 ldlm: Use interval tree to update kms")
Reviewed-on: https://review.whamcloud.com/20779
--- fs/lustre/ldlm/ldlm_extent.c | 66 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 51 insertions(+), 15 deletions(-)
fs/lustre/ldlm/ldlm_extent.c +51 -15
@@ -55,13 +55,22 @@#include "ldlm_internal.h" #include <linux/interval_tree_generic.h> -#define START (node) ((node)->l_policy_data.l_extent.start) -#define LAST (node) ((node)->l_policy_data.l_extent.end) +/* We sort the interval tree in reverse order, because we sometimes +* and to find the interval with the highest end, and the first/next +* iteration only allows is to walk in increasing order of start. +*/ +#define ISTART (end) (U64_MAX - (end)) +#define IEND (start) (U64_MAX - (start)) + +#define START (node) ISTART((node)->l_policy_data.l_extent.end) +#define LAST (node) IEND((node)->l_policy_data.l_extent.start) INTERVAL_TREE_DEFINE(struct ldlm_lock, l_rb, u64, __subtree_last, START, LAST, static, extent);/* When a lock is cancelled by a client, the KMS may undergo change if this -* is the "highest lock". This function returns the new KMS value. +* is the "highest lock". This function returns the new KMS value, updating +* it only if we were the highest lock. +* * Caller must hold lr_lock already. * * NB: A lock on [x,y] protects a KMS of up to y + 1 bytes!
@@ -69,8 +78,11 @@u64 ldlm_extent_shift_kms (struct ldlm_lock *lock,u64 old_kms) {struct ldlm_resource *res = lock->l_resource; +struct ldlm_interval_tree *tree;struct ldlm_lock *lck;u64 kms =0 ; +int idx =0 ; +bool complete;/* don't let another thread in ldlm_extent_shift_kms race in * just after we finish and take our lock into account in its
@@ -78,20 +90,44 @@ u64 ldlm_extent_shift_kms(struct ldlm_lock *lock, u64 old_kms)*/ ldlm_set_kms_ignore (lock); -list_for_each_entry (lck, &res->lr_granted, l_res_link) { -if (ldlm_is_kms_ignore (lck)) -continue ; +/* We iterate over the lock trees, looking for the largest kms +* smaller than the current one. Note that each tree is +* iterated starting a largest end, because the interval tree +* is stored last-to-first order. +*/ +for (idx =0 ; idx <LCK_MODE_NUM ; idx++) { + tree = &res->lr_itree[idx]; -if (lck->l_policy_data.l_extent.end >= old_kms) -return old_kms; +for (lck =extent_iter_first (&tree->lit_root,0 ,U64_MAX ); + lck; + lck =extent_iter_next (lck,0 ,U64_MAX )) { +if (ldlm_is_kms_ignore (lck)) +continue ; -/* This extent _has_ to be smaller than old_kms (checked above) -* so kms can only ever be smaller or the same as old_kms. +/* This is the last lock-end that doesn't ignore +* kms. +* If it has a greater or equal kms, we are not +* the highest lock (or we share that distinction +* with another lock), and don't need to update KMS. +* Record old_kms and stop looking. +*/ +if (lck->l_policy_data.l_extent.end >= old_kms) { + kms = old_kms; + complete = true; + }else + kms = lck->l_policy_data.l_extent.end +1 ; +break ; + } + +/* this tells us we're not the highest lock, so we don't need +* to check the remaining trees */ -if (lck->l_policy_data.l_extent.end +1 > kms) - kms = lck->l_policy_data.l_extent.end +1 ; +if (complete) +break ; } -LASSERTF (kms <= old_kms,"kms %llu old_kms %llu\n" , kms, old_kms); + +LASSERTF (kms <= old_kms,"kms %llu old_kms %llu\n" , kms, + old_kms);return kms; }
@@ -197,9 +233,9 @@ void ldlm_extent_search(struct rb_root_cached *root, {struct ldlm_lock *lock; -for (lock =extent_iter_first (root, start, end); +for (lock =extent_iter_first (root,ISTART (end),IEND (start)); lock; - lock =extent_iter_next (lock, start, end)) + lock =extent_iter_next (lock,ISTART (end),IEND (start)))if (matches (lock, data))break ; }
Signature · 1 line
1.8.3.1
PATCH 18/151 lustre: osc: prepare OSC code to be used from MDC
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Move osc/osc_cl_include.h to include/lustre_osc.h
to be included from both OSC and MDC.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 036d987c6d71 ("LU-3285 osc: prepare OSC code to be used from MDC")
Reviewed-on: https://review.whamcloud.com/22996
---
.../osc_cl_internal.h => include/lustre_osc.h} | 151 ++++++++++++++++-----
fs/lustre/osc/lproc_osc.c | 2 +
fs/lustre/osc/osc_cache.c | 3 +-
fs/lustre/osc/osc_dev.c | 14 +-
fs/lustre/osc/osc_internal.h | 107 +++------------
fs/lustre/osc/osc_io.c | 18 +--
fs/lustre/osc/osc_lock.c | 3 +-
fs/lustre/osc/osc_object.c | 20 +--
fs/lustre/osc/osc_page.c | 4 +-
fs/lustre/osc/osc_quota.c | 2 +
fs/lustre/osc/osc_request.c | 3 +-
11 files changed, 157 insertions(+), 170 deletions(-)
rename fs/lustre/{osc/osc_cl_internal.h => include/lustre_osc.h} (86%)fs/lustre/include/lustre_osc.h renamed from fs/lustre/osc/osc_cl_internal.h +118 -33
@@ -31,22 +31,97 @@* Lustre is a trademark of Sun Microsystems, Inc. */ /* -* This file is part of Lustre, http://www.lustre.org/ -* Lustre is a trademark of Sun Microsystems, Inc. +* fs/lustre/include/lustre_osc.h * -* Internal interfaces of OSC layer. +* OSC layer structures and methods common for both OSC and MDC. +* +* This file contains OSC interfaces used by OSC and MDC. Most of them +* were just moved from fs/lustre/osc/osc_cl_internal.h for Data-on-MDT +* purposes * * Author: Nikita Danilov <[email protected]> -* Author: Jinshan Xiong <[email protected]> +* Author: Jinshan Xiong <[email protected]> +* Author: Mikhail Pershin <[email protected]> */ -#ifndef OSC_CL_INTERNAL_H -#define OSC_CL_INTERNAL_H +#ifndef LUSTRE_OSC_H +#define LUSTRE_OSC_H + +#include <linux/rhashtable.h> #include <obd.h> -/* osc_build_res_name() */ #include <cl_object.h> -#include "osc_internal.h" + +struct osc_quota_info { +/* linkage for quota hash table */ +struct rhash_head oqi_hash; +u32 oqi_id; +struct rcu_head rcu; +}; + +enum async_flags { +ASYNC_READY =0x1 ,/* osc_make_ready will not be called +* before this page is added to an rpc +*/ +ASYNC_URGENT =0x2 ,/* page must be put into an RPC before +* return +*/ +ASYNC_COUNT_STABLE =0x4 ,/* osc_refresh_count will not be called +* to give the caller a chance to update +* or cancel the size of the io +*/ +ASYNC_HP =0x10 , +}; + +struct osc_async_page { +int oap_magic; +unsigned short oap_cmd; + +struct list_head oap_pending_item; +struct list_head oap_rpc_item; + +u64 oap_obj_off; +unsigned int oap_page_off; +enum async_flags oap_async_flags; + +struct brw_page oap_brw_page; + +struct ptlrpc_request *oap_request; +struct client_obd *oap_cli; +struct osc_object *oap_obj; + +spinlock_t oap_lock; +}; + +#define oap_page oap_brw_page.pg +#define oap_count oap_brw_page.count +#define oap_brw_flags oap_brw_page.flag + +static inline struct osc_async_page *brw_page2oap (struct brw_page *pga) +{ + return container_of(pga, struct osc_async_page, oap_brw_page); +} + +static inline void osc_wake_cache_waiters (struct client_obd *cli) +{ +wake_up (&cli->cl_cache_waiters); +} + +struct osc_device { +struct cl_device od_cl; +struct obd_export *od_exp; + +/* Write stats is actually protected by client_obd's lock. */ +struct osc_stats { +u64 os_lockless_writes;/* by bytes */ +u64 os_lockless_reads;/* by bytes */ +u64 os_lockless_truncates;/* by times */ + } od_stats; + +/* configuration item(s) */ +int od_contention_time; +int od_lockless_truncate; +};/* \defgroup osc osc * @{
@@ -166,6 +241,8 @@ struct osc_object {/* number of active IOs of this object */ atomic_t oo_nr_ios;wait_queue_head_t oo_io_waitq; + +bool oo_initialized; };static inline void osc_object_lock (struct osc_object *obj)
@@ -353,21 +430,13 @@ struct osc_page {extern struct kmem_cache *osc_thread_kmem;extern struct kmem_cache *osc_session_kmem;extern struct kmem_cache *osc_extent_kmem; +extern struct kmem_cache *osc_quota_kmem; -extern struct lu_device_type osc_device_type;extern struct lu_context_key osc_key;extern struct lu_context_key osc_session_key;#define OSC_FLAGS (ASYNC_URGENT | ASYNC_READY) -int osc_lock_init (const struct lu_env *env, -struct cl_object *obj,struct cl_lock *lock, -const struct cl_io *io); -int osc_io_init (const struct lu_env *env, -struct cl_object *obj,struct cl_io *io); -struct lu_object *osc_object_alloc (const struct lu_env *env, -const struct lu_object_header *hdr, -struct lu_device *dev);int osc_page_init (const struct lu_env *env,struct cl_object *obj,struct cl_page *page,pgoff_t ind);
@@ -413,7 +482,7 @@ void osc_io_unplug(const struct lu_env *env, struct client_obd *cli,/**************************************************************************** * -* Accessors. +* Accessors and type conversions. * */
@@ -440,14 +509,8 @@ static inline struct osc_io *osc_env_io(const struct lu_env *env)return &osc_env_session (env)->os_io; } -static inline int osc_is_object (const struct lu_object *obj) -{ -return obj->lo_dev->ld_type == &osc_device_type; -} -static inline struct osc_device *lu2osc_dev (const struct lu_device *d) { -LINVRNT (d->ld_type == &osc_device_type); return container_of(d, struct osc_device, od_cl.cd_lu_dev); }
@@ -463,7 +526,6 @@ static inline struct client_obd *osc_cli(const struct osc_object *obj)static inline struct osc_object *cl2osc (const struct cl_object *obj) { -LINVRNT (osc_is_object (&obj->co_lu)); return container_of(obj, struct osc_object, oo_cl); }
@@ -472,6 +534,36 @@ static inline struct cl_object *osc2cl(const struct osc_object *obj)return (struct cl_object *)&obj->oo_cl; } +static inline struct osc_device *obd2osc_dev (const struct obd_device *d) +{ + return container_of_safe(d->obd_lu_dev, struct osc_device, + od_cl.cd_lu_dev); +} + +static inline struct lu_device *osc2lu_dev (struct osc_device *osc) +{ +return &osc->od_cl.cd_lu_dev; +} + +static inline struct lu_object *osc2lu (struct osc_object *osc) +{ +return &osc->oo_cl.co_lu; +} + +static inline struct osc_object *lu2osc (const struct lu_object *obj) +{ + return container_of_safe(obj, struct osc_object, oo_cl.co_lu); +} + +static inline struct osc_io *cl2osc_io (const struct lu_env *env, +const struct cl_io_slice *slice) +{ + struct osc_io *oio = container_of_safe(slice, struct osc_io, oi_cl); + +LINVRNT (oio ==osc_env_io (env)); +return oio; +} +static inline enum ldlm_mode osc_cl_lock2ldlm (enum cl_lock_mode mode) {LASSERT (mode ==CLM_READ || mode ==CLM_WRITE || mode ==CLM_GROUP );
@@ -496,7 +588,6 @@ static inline enum cl_lock_mode osc_ldlm2cl_lock(enum ldlm_mode mode)static inline struct osc_page *cl2osc_page (const struct cl_page_slice *slice) { -LINVRNT (osc_is_object (&slice->cpl_obj->co_lu)); return container_of(slice, struct osc_page, ops_cl); }
@@ -532,15 +623,9 @@ static inline struct osc_page *oap2osc_page(struct osc_async_page *oap)static inline struct osc_lock *cl2osc_lock (const struct cl_lock_slice *slice) { -LINVRNT (osc_is_object (&slice->cls_obj->co_lu)); return container_of(slice, struct osc_lock, ols_cl); } -static inline struct osc_lock *osc_lock_at (const struct cl_lock *lock) -{ -return cl2osc_lock (cl_lock_at (lock, &osc_device_type)); -} -static inline int osc_io_srvlock (struct osc_io *oio) {return (oio->oi_lockless && !oio->oi_cl.cis_io->ci_no_srvlock);
@@ -667,4 +752,4 @@ bool osc_page_gang_lookup(const struct lu_env *env, struct cl_io *io, osc_page_gang_cbt cb, void *cbdata);/* @} osc */ -#endif /* OSC_CL_INTERNAL_H */ +#endif /* LUSTRE_OSC_H */
fs/lustre/osc/lproc_osc.c +2 -0
@@ -38,6 +38,8 @@#include <obd_class.h> #include <lprocfs_status.h> #include <linux/seq_file.h> +#include <lustre_osc.h> +#include "osc_internal.h" static ssize_t active_show(struct kobject *kobj, struct attribute *attr,
fs/lustre/osc/osc_cache.c +2 -1
@@ -38,7 +38,8 @@#define DEBUG_SUBSYSTEM S_OSC -#include "osc_cl_internal.h" +#include <lustre_osc.h> +#include "osc_internal.h" static int extent_debug;/* set it to be true for more debug */
fs/lustre/osc/osc_dev.c +2 -12
@@ -39,8 +39,9 @@/* class_name2obd() */ #include <obd_class.h> +#include <lustre_osc.h> -#include "osc_cl_internal.h" +#include "osc_internal.h" /** \addtogroup osc * @{
@@ -91,17 +92,6 @@ struct lu_kmem_descr osc_caches[] = {/***************************************************************************** * -* Type conversions. -* -*/ - -static struct lu_device *osc2lu_dev (struct osc_device *osc) -{ -return &osc->od_cl.cd_lu_dev; -} - -/***************************************************************************** -* * Osc device and device type functions. * */
fs/lustre/osc/osc_internal.h +20 -87
134 lines not on this page · raw message
fs/lustre/osc/osc_io.c +2 -16
@@ -39,8 +39,9 @@#define DEBUG_SUBSYSTEM S_OSC#include <lustre_obdo.h> +#include <lustre_osc.h> -#include "osc_cl_internal.h" +#include "osc_internal.h" /** \addtogroup osc * @{
@@ -48,21 +49,6 @@/***************************************************************************** * -* Type conversions. -* -*/ - -static struct osc_io *cl2osc_io (const struct lu_env *env, -const struct cl_io_slice *slice) -{ - struct osc_io *oio = container_of_safe(slice, struct osc_io, oi_cl); - -LINVRNT (oio ==osc_env_io (env)); -return oio; -} - -/***************************************************************************** -* * io operations. * */
fs/lustre/osc/osc_lock.c +2 -1
@@ -40,8 +40,9 @@/* fid_build_reg_res_name() */ #include <lustre_fid.h> +#include <lustre_osc.h> -#include "osc_cl_internal.h" +#include "osc_internal.h" /** \addtogroup osc * @{
fs/lustre/osc/osc_object.c +2 -18
@@ -37,8 +37,9 @@*/ #define DEBUG_SUBSYSTEM S_OSC +#include <lustre_osc.h> -#include "osc_cl_internal.h" +#include "osc_internal.h" /** \addtogroup osc * @{
@@ -46,23 +47,6 @@/***************************************************************************** * -* Type conversions. -* -*/ - -static struct lu_object *osc2lu (struct osc_object *osc) -{ -return &osc->oo_cl.co_lu; -} - -static struct osc_object *lu2osc (const struct lu_object *obj) -{ -LINVRNT (osc_is_object (obj)); - return container_of(obj, struct osc_object, oo_cl.co_lu); -} - -/***************************************************************************** -* * Object operations. * */
fs/lustre/osc/osc_page.c +3 -1
@@ -39,7 +39,9 @@#define DEBUG_SUBSYSTEM S_OSC#include <linux/math64.h> -#include "osc_cl_internal.h" +#include <lustre_osc.h> + +#include "osc_internal.h" static void osc_lru_del (struct client_obd *cli,struct osc_page *opg);static void osc_lru_use (struct client_obd *cli,struct osc_page *opg);
fs/lustre/osc/osc_quota.c +2 -0
@@ -25,6 +25,8 @@*/ #include <obd_class.h> +#include <lustre_osc.h> +#include "osc_internal.h" static const struct rhashtable_params quota_hash_params = {
fs/lustre/osc/osc_request.c +2 -1
@@ -50,8 +50,9 @@#include <lustre_fid.h> #include <obd_class.h> #include <obd.h> +#include <lustre_osc.h> +#include "osc_internal.h" -#include "osc_cl_internal.h" atomic_t osc_pool_req_count;unsigned int osc_reqpool_maxreqcount;
Signature · 1 line
1.8.3.1
PATCH 19/151 lustre: statahead: support striped directory
In reply to James SimmonsFrom: Lai Siyao <[email protected]>
Current statahead will getattr by fid, but for striped directory, it needs to fix its parent fid to stripe fid, which needs file name, so pack file name in md_op_data.
Remove check for remote dir check in lmv_intent_getattr_async(), which is not necessary, and not checking it won't cause anything wrong, while the code is simpler.
WC-bug-id: https://jira.whamcloud.com/browse/LU-5106
Lustre-commit: 9e7952c045a3 ("LU-5106 statahead: support striped directory")
Reviewed-on: https://review.whamcloud.com/27683
--- fs/lustre/llite/statahead.c | 3 ++- fs/lustre/lmv/lmv_obd.c | 27 +++++++++++---------------- 2 files changed, 13 insertions(+), 17 deletions(-)
fs/lustre/llite/statahead.c +2 -1
@@ -749,7 +749,8 @@ static void sa_fini_data(struct md_enqueue_info *minfo)if (!minfo)return ERR_PTR (-ENOMEM ); - op_data =ll_prep_md_op_data (&minfo->mi_data, dir, child,NULL ,0 ,0 , + op_data =ll_prep_md_op_data (&minfo->mi_data, dir, child, + entry->se_qstr.name, entry->se_qstr.len,0 ,LUSTRE_OPC_ANY ,NULL );if (IS_ERR (op_data)) {kfree (minfo);
fs/lustre/lmv/lmv_obd.c +11 -16
@@ -2904,30 +2904,25 @@ static int lmv_intent_getattr_async(struct obd_export *exp,struct md_op_data *op_data = &minfo->mi_data;struct obd_device *obd = exp->exp_obd;struct lmv_obd *lmv = &obd->u.lmv; -struct lmv_tgt_desc *ptgt =NULL ; -struct lmv_tgt_desc *ctgt =NULL ; +struct lmv_tgt_desc *tgt =NULL ;if (!fid_is_sane (&op_data->op_fid2))return -EINVAL ; - ptgt =lmv_locate_mds (lmv, op_data, &op_data->op_fid1); -if (IS_ERR (ptgt)) -return PTR_ERR (ptgt); - - ctgt =lmv_locate_mds (lmv, op_data, &op_data->op_fid2); -if (IS_ERR (ctgt)) -return PTR_ERR (ctgt); + tgt =lmv_locate_mds (lmv, op_data, &op_data->op_fid1); +if (IS_ERR (tgt)) +return PTR_ERR (tgt);/* -* if child is on remote MDT, we need 2 async RPCs to fetch both LOOKUP -* lock on parent, and UPDATE lock on child MDT, which makes all -* complicated. Considering remote dir is rare case, and not supporting -* it in statahead won't cause any issue, drop its support for now. +* no special handle for remote dir, which needs to fetch both LOOKUP +* lock on parent, and then UPDATE lock on child MDT, which makes all +* complicated because this is done async. So only LOOKUP lock is +* fetched for remote dir, but considering remote dir is rare case, +* and not supporting it in statahead won't cause any issue, just leave +* it as is. */ -if (ptgt != ctgt) -return -ENOTSUPP ; -return md_intent_getattr_async (ptgt->ltd_exp, minfo); +return md_intent_getattr_async (tgt->ltd_exp, minfo); } static int lmv_revalidate_lock(struct obd_export *exp, struct lookup_intent *it,
Signature · 1 line
1.8.3.1
PATCH 20/151 lustre: readdir: improve striped readdir
In reply to James SimmonsFrom: Lai Siyao <[email protected]>
Striped directory needs to build its directory page from sub-stripe directory pages, current code iterate sub stripe directory pages to search dirent for each hash, this is inefficient, as may cause statahead fail because statahead thread is slow in readdir, while 'ls' is faster and can't find cached statahead entries, and finally cause statahead fail.
This patch introduces a struct lmv_dir_ctxt which saves dir page and current dirent for all stripes, to find the dirent which has the closest hash value it only needs to compare the dirent of all stripes, and then pop this dirent from its stripe, until all stripes reache the end.
This patch contains another fix: previously LDP_COLLIDE is set by default, change to set dir page end hash 'ldp_hash_end' to hash of next dirent, and only set LDP_COLLIDE when 'ldp_hash_end' equals last dirent hash 'lde_hash'. We should avoid dir hash collision.
WC-bug-id: https://jira.whamcloud.com/browse/LU-5106
Lustre-commit: 98fc9a77446a ("LU-5106 readdir: improve striped readdir")
Reviewed-on: https://review.whamcloud.com/27663
--- fs/lustre/lmv/lmv_obd.c | 423 ++++++++++++++++++++++++++---------------------- 1 file changed, 231 insertions(+), 192 deletions(-)
fs/lustre/lmv/lmv_obd.c +231 -192
@@ -2001,60 +2001,79 @@ static int lmv_fsync(struct obd_export *exp, const struct lu_fid *fid,return md_fsync (tgt->ltd_exp, fid, request); } -/** -* Get current minimum entry from striped directory -* -* This function will search the dir entry, whose hash value is the -* closest(>=) to @hash_offset, from all of sub-stripes, and it is -* only being called for striped directory. -* -* @exp: export of LMV -* @op_data: parameters transferred beween client MD stack -* stripe_information will be included in this -* parameter -* @cb_op: ldlm callback being used in enqueue in -* mdc_read_page -* @hash_offset: the hash value, which is used to locate -* minum(closet) dir entry -* @stripe_offset: the caller use this to indicate the stripe -* index of last entry, so to avoid hash conflict -* between stripes. It will also be used to -* return the stripe index of current dir entry. -* @entp: the minum entry and it also is being used -* to input the last dir entry to resolve the -* hash conflict -* -* @ppage: the page which holds the minum entry -* -* Return: = 0 get the entry successfully -* negative errno (< 0) does not get the entry -*/ -static int lmv_get_min_striped_entry (struct obd_export *exp, -struct md_op_data *op_data, -struct md_callback *cb_op, -u64 hash_offset,int *stripe_offset, -struct lu_dirent **entp, -struct page **ppage) +struct stripe_dirent { +struct page *sd_page; +struct lu_dirpage *sd_dp; +struct lu_dirent *sd_ent; +bool sd_eof; +}; + +struct lmv_dir_ctxt { +struct lmv_obd *ldc_lmv; +struct md_op_data *ldc_op_data; +struct md_callback *ldc_cb_op; +u64 ldc_hash; +int ldc_count; +struct stripe_dirent ldc_stripes[0 ]; +}; + +static inline void put_stripe_dirent (struct stripe_dirent *stripe) +{ +if (stripe->sd_page) { +kunmap (stripe->sd_page); +put_page (stripe->sd_page); + stripe->sd_page =NULL ; + } +} + +static inline void put_lmv_dir_ctxt (struct lmv_dir_ctxt *ctxt) { -struct lmv_stripe_md *lsm = op_data->op_mea1; -struct obd_device *obd = exp->exp_obd; -struct lmv_obd *lmv = &obd->u.lmv; -struct lu_dirent *min_ent =NULL ; -struct page *min_page =NULL ; -struct lmv_tgt_desc *tgt; -int stripe_count; -int min_idx =0 ; -int rc =0 ;int i; - stripe_count = lsm->lsm_md_stripe_count; -for (i =0 ; i < stripe_count; i++) { -u64 stripe_hash = hash_offset; -struct lu_dirent *ent =NULL ; -struct page *page =NULL ; -struct lu_dirpage *dp; +for (i =0 ; i < ctxt->ldc_count; i++) +put_stripe_dirent (&ctxt->ldc_stripes[i]); +} + +static struct lu_dirent *stripe_dirent_next (struct lmv_dir_ctxt *ctxt, +struct stripe_dirent *stripe, +int stripe_index) +{ +struct lu_dirent *ent = stripe->sd_ent; +u64 hash = ctxt->ldc_hash; +u64 end; +int rc =0 ; + +LASSERT (stripe == &ctxt->ldc_stripes[stripe_index]); + +if (ent) { + ent =lu_dirent_next (ent); +if (!ent) { +check_eof: + end =le64_to_cpu (stripe->sd_dp->ldp_hash_end); + +put_stripe_dirent (stripe); + +if (end ==MDS_DIR_END_OFF ) { + stripe->sd_ent =NULL ; + stripe->sd_eof = true; +return NULL ; + } +LASSERT (hash <= end); + hash = end; + } + } + +if (!ent) { +struct md_op_data *op_data = ctxt->ldc_op_data; +struct lmv_oinfo *oinfo; +struct lu_fid fid = op_data->op_fid1; +struct inode *inode = op_data->op_data; +struct lmv_tgt_desc *tgt; - tgt =lmv_get_target (lmv, lsm->lsm_md_oinfo[i].lmo_mds,NULL ); +LASSERT (!stripe->sd_page); + + oinfo = &op_data->op_mea1->lsm_md_oinfo[stripe_index]; + tgt =lmv_get_target (ctxt->ldc_lmv, oinfo->lmo_mds,NULL );if (IS_ERR (tgt)) { rc =PTR_ERR (tgt); goto out;
@@ -2064,87 +2083,117 @@ static int lmv_get_min_striped_entry(struct obd_export *exp,* op_data will be shared by each stripe, so we need * reset these value for each stripe */ - op_data->op_fid1 = lsm->lsm_md_oinfo[i].lmo_fid; - op_data->op_fid2 = lsm->lsm_md_oinfo[i].lmo_fid; - op_data->op_data = lsm->lsm_md_oinfo[i].lmo_root; -next: - rc =md_read_page (tgt->ltd_exp, op_data, cb_op, stripe_hash, - &page); + op_data->op_fid1 = oinfo->lmo_fid; + op_data->op_fid2 = oinfo->lmo_fid; + op_data->op_data = oinfo->lmo_root; + + rc =md_read_page (tgt->ltd_exp, op_data, ctxt->ldc_cb_op, hash, + &stripe->sd_page); + + op_data->op_fid1 = fid; + op_data->op_fid2 = fid; + op_data->op_data = inode; +if (rc) goto out; - dp =page_address (page); -for (ent =lu_dirent_start (dp); ent; - ent =lu_dirent_next (ent)) { -/* Skip dummy entry */ -if (!le16_to_cpu (ent->lde_namelen)) -continue ; + stripe->sd_dp =page_address (stripe->sd_page); + ent =lu_dirent_start (stripe->sd_dp); + } -if (le64_to_cpu (ent->lde_hash) < hash_offset) -continue ; +for (; ent; ent =lu_dirent_next (ent)) { +/* Skip dummy entry */ +if (!le16_to_cpu (ent->lde_namelen)) +continue ; -if (le64_to_cpu (ent->lde_hash) == hash_offset && - (*entp == ent || i < *stripe_offset)) -continue ; +/* skip . and .. for other stripes */ +if (stripe_index && + (strncmp (ent->lde_name,"." , +le16_to_cpu (ent->lde_namelen)) ==0 || +strncmp (ent->lde_name,".." , +le16_to_cpu (ent->lde_namelen)) ==0 )) +continue ; -/* skip . and .. for other stripes */ -if (i && (!strncmp (ent->lde_name,"." , -le16_to_cpu (ent->lde_namelen)) || - !strncmp (ent->lde_name,".." , -le16_to_cpu (ent->lde_namelen)))) -continue ; -break ; - } +if (le64_to_cpu (ent->lde_hash) < hash) +continue ; -if (!ent) { - stripe_hash =le64_to_cpu (dp->ldp_hash_end); +break ; + } -kunmap (page); -put_page (page); - page =NULL ; +if (!ent) + goto check_eof; -/* -* reach the end of current stripe, go to next stripe -*/ -if (stripe_hash ==MDS_DIR_END_OFF ) +out: + stripe->sd_ent = ent; +/* treat error as eof, so dir can be partially accessed */ +if (rc) { +put_stripe_dirent (stripe); + stripe->sd_eof = true; +LCONSOLE_WARN ("dir " DFID " stripe %d readdir failed: %d, directory is partially accessed!\n" , +PFID (&ctxt->ldc_op_data->op_fid1), stripe_index, + rc); + } +return ent; +} + +/** +* Get dirent with the closest hash for striped directory +* +* This function will search the dir entry, whose hash value is the +* closest(>=) to hash from all of sub-stripes, and it is only being called +* for striped directory. +* +* @param ctxt dir read context +* +* Return: dirent get the entry successfully NULL does not get +* the entry, normally it means it reaches the end of +* the directory, while read stripe dirent error is +* ignored to allow partial access. +*/ +static struct lu_dirent *lmv_dirent_next (struct lmv_dir_ctxt *ctxt) +{ +struct stripe_dirent *stripe; +struct lu_dirent *ent =NULL ; +int i; +int min =-1 ; + +/* TODO: optimize with k-way merge sort */ +for (i =0 ; i < ctxt->ldc_count; i++) { + stripe = &ctxt->ldc_stripes[i]; +if (stripe->sd_eof) +continue ; + +if (!stripe->sd_ent) { +/* locate starting entry */ +stripe_dirent_next (ctxt, stripe, i); +if (!stripe->sd_ent) { +LASSERT (stripe->sd_eof);continue ; -else - goto next; + } } -if (min_ent) { -if (le64_to_cpu (min_ent->lde_hash) > -le64_to_cpu (ent->lde_hash)) { - min_ent = ent; -kunmap (min_page); -put_page (min_page); - min_idx = i; - min_page = page; - }else { -kunmap (page); -put_page (page); - page =NULL ; - } - }else { - min_ent = ent; - min_page = page; - min_idx = i; +if (min ==-1 || +le64_to_cpu (ctxt->ldc_stripes[min].sd_ent->lde_hash) > +le64_to_cpu (stripe->sd_ent->lde_hash)) { + min = i; +if (le64_to_cpu (stripe->sd_ent->lde_hash) == + ctxt->ldc_hash) +break ; } } -out: -if (*ppage) { -kunmap (*ppage); -put_page (*ppage); +if (min !=-1 ) { + stripe = &ctxt->ldc_stripes[min]; + ent = stripe->sd_ent; +/* pop found dirent */ +stripe_dirent_next (ctxt, stripe, min); } - *stripe_offset = min_idx; - *entp = min_ent; - *ppage = min_page; -return rc; + +return ent; }/** -* Build dir entry page from a striped directory +* Build dir entry page for striped directory * * This function gets one entry by @offset from a striped directory. It will * read entries from all of stripes, and choose one closest to the required
@@ -2153,12 +2202,11 @@ static int lmv_get_min_striped_entry(struct obd_export *exp,* and .. in a directory. * 2. op_data will be shared by all of stripes, instead of allocating new * one, so need to restore before reusing. -* 3. release the entry page if that is not being chosen. * * @exp: obd export refer to LMV * @op_data: hold those MD parameters of read_entry * @cb_op: ldlm callback being used in enqueue in mdc_read_entry -* @ldp: the entry being read +* @offset: the entry being read * @ppage: the page holding the entry. Note: because the entry * will be accessed in upper layer, so we need hold the * page until the usages of entry is finished, see
@@ -2167,81 +2215,80 @@ static int lmv_get_min_striped_entry(struct obd_export *exp,* Returns: =0 if get entry successfully * <0 cannot get entry */ -static int lmv_read_striped_page (struct obd_export *exp, +static int lmv_striped_read_page (struct obd_export *exp,struct md_op_data *op_data,struct md_callback *cb_op,u64 offset,struct page **ppage) { -struct inode *master_inode = op_data->op_data; -struct lu_fid master_fid = op_data->op_fid1; -u64 hash_offset = offset; -u32 ldp_flags; -struct page *min_ent_page =NULL ; -struct page *ent_page =NULL ; -struct lu_dirent *min_ent =NULL ; -struct lu_dirent *last_ent; -struct lu_dirent *ent; +struct page *page =NULL ;struct lu_dirpage *dp; +void *start; +struct lu_dirent *ent; +struct lu_dirent *last_ent; +int stripe_count; +struct lmv_dir_ctxt *ctxt; +struct lu_dirent *next =NULL ; +u16 ent_size;size_t left_bytes; -int ent_idx =0 ; -void *area; -int rc; +int rc =0 ;/* * Allocate a page and read entries from all of stripes and fill * the page by hash order */ - ent_page =alloc_page (GFP_KERNEL ); -if (!ent_page) + page =alloc_page (GFP_KERNEL ); +if (!page)return -ENOMEM ;/* Initialize the entry page */ - dp =kmap (ent_page); + dp =kmap (page);memset (dp,0 ,sizeof (*dp)); dp->ldp_hash_start =cpu_to_le64 (offset); - ldp_flags =LDF_COLLIDE ; - area = dp +1 ; + start = dp +1 ; left_bytes =PAGE_SIZE -sizeof (*dp); - ent = area; + ent = start; last_ent = ent; - do { - u16 ent_size; - - /* Find the minum entry from all sub-stripes */ - rc = lmv_get_min_striped_entry(exp, op_data, cb_op, hash_offset, - &ent_idx, &min_ent, - &min_ent_page); - if (rc) - goto out; - /* - * If it can not get minum entry, it means it already reaches - * the end of this directory - */ - if (!min_ent) { - last_ent->lde_reclen = 0; - hash_offset = MDS_DIR_END_OFF; - goto out; +/* initialize dir read context */ + stripe_count = op_data->op_mea1->lsm_md_stripe_count; + ctxt = kzalloc(offsetof(typeof(*ctxt), ldc_stripes[stripe_count]), + GFP_NOFS); +if (!ctxt) { + rc = -ENOMEM ; + goto free_page; + } + ctxt->ldc_lmv = &exp->exp_obd->u.lmv; + ctxt->ldc_op_data = op_data; + ctxt->ldc_cb_op = cb_op; + ctxt->ldc_hash = offset; + ctxt->ldc_count = stripe_count; + +while (1 ) { + next =lmv_dirent_next (ctxt); + +/* end of directory */ +if (!next) { + ctxt->ldc_hash =MDS_DIR_END_OFF ; +break ; } + ctxt->ldc_hash =le64_to_cpu (next->lde_hash); - ent_size = le16_to_cpu(min_ent->lde_reclen); + ent_size =le16_to_cpu (next->lde_reclen);/* - * the last entry lde_reclen is 0, but it might not - * the end of this entry of this temporay entry +* the last entry lde_reclen is 0, but it might not be the last +* one of this temporay dir page */ if (!ent_size) ent_size =lu_dirent_calc_size ( - le16_to_cpu(min_ent->lde_namelen), - le32_to_cpu(min_ent->lde_attrs)); - if (ent_size > left_bytes) { - last_ent->lde_reclen = cpu_to_le16(0); - hash_offset = le64_to_cpu(min_ent->lde_hash); - goto out; - } +le16_to_cpu (next->lde_namelen), +le32_to_cpu (next->lde_attrs)); +/* page full */ +if (ent_size > left_bytes) +break ; - memcpy(ent, min_ent, ent_size); +memcpy (ent, next, ent_size);/* * Replace . with master FID and Replace .. with the parent FID
@@ -2250,49 +2297,41 @@ static int lmv_read_striped_page(struct obd_export *exp,if (!strncmp (ent->lde_name,"." ,le16_to_cpu (ent->lde_namelen)) &&le16_to_cpu (ent->lde_namelen) ==1 ) -fid_cpu_to_le (&ent->lde_fid, &master_fid); +fid_cpu_to_le (&ent->lde_fid, &op_data->op_fid1);else if (!strncmp (ent->lde_name,".." ,le16_to_cpu (ent->lde_namelen)) &&le16_to_cpu (ent->lde_namelen) ==2 )fid_cpu_to_le (&ent->lde_fid, &op_data->op_fid3); +CDEBUG (D_INODE ,"entry %.*s hash %#llx\n" , +le16_to_cpu (ent->lde_namelen), ent->lde_name, +le64_to_cpu (ent->lde_hash)); + left_bytes -= ent_size; ent->lde_reclen =cpu_to_le16 (ent_size); last_ent = ent; ent = (void *)ent + ent_size; - hash_offset =le64_to_cpu (min_ent->lde_hash); -if (hash_offset ==MDS_DIR_END_OFF ) { - last_ent->lde_reclen =0 ; -break ; - } - }while (1 ); -out: -if (min_ent_page) { -kunmap (min_ent_page); -put_page (min_ent_page); } -if (unlikely (rc)) { -__free_page (ent_page); - ent_page =NULL ; - }else { -if (ent == area) - ldp_flags |=LDF_EMPTY ; - dp->ldp_flags |=cpu_to_le32 (ldp_flags); - dp->ldp_hash_end =cpu_to_le64 (hash_offset); - } + last_ent->lde_reclen =0 ; -/* -* We do not want to allocate md_op_data during each -* dir entry reading, so op_data will be shared by every stripe, -* then we need to restore it back to original value before -* return to the upper layer -*/ - op_data->op_fid1 = master_fid; - op_data->op_fid2 = master_fid; - op_data->op_data = master_inode; +if (ent == start) + dp->ldp_flags |=LDF_EMPTY ; +else if (ctxt->ldc_hash ==le64_to_cpu (last_ent->lde_hash)) + dp->ldp_flags |=LDF_COLLIDE ; + dp->ldp_flags =cpu_to_le32 (dp->ldp_flags); + dp->ldp_hash_end =cpu_to_le64 (ctxt->ldc_hash); + +put_lmv_dir_ctxt (ctxt); +kfree (ctxt); + + *ppage = page; + +return 0 ; - *ppage = ent_page; +free_page: +kunmap (page); +__free_page (page);return rc; }
@@ -2307,7 +2346,7 @@ static int lmv_read_page(struct obd_export *exp, struct md_op_data *op_data,struct lmv_tgt_desc *tgt;if (unlikely (lsm)) { -return lmv_read_striped_page (exp, op_data, cb_op, +return lmv_striped_read_page (exp, op_data, cb_op, offset, ppage); }
Signature · 1 line
1.8.3.1
PATCH 21/151 lustre: llog: consolidate common error checking
In reply to James SimmonsFrom: Andreas Dilger <[email protected]>
Consolidate error checking for corrupted and empty llog files detected when the llog file is being opened into a single routine llog_cat_process_common().
WC-bug-id: https://jira.whamcloud.com/browse/LU-9153
Lustre-commit: b522f7b4b1de ("LU-9153 llog: consolidate common error checking")
Reviewed-on: https://review.whamcloud.com/26211
--- fs/lustre/obdclass/llog_cat.c | 48 ++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 12 deletions(-)
fs/lustre/obdclass/llog_cat.c +36 -12
@@ -140,32 +140,55 @@ int llog_cat_close(const struct lu_env *env, struct llog_handle *cathandle) }EXPORT_SYMBOL (llog_cat_close); -static int llog_cat_process_cb (const struct lu_env *env, -struct llog_handle *cat_llh, -struct llog_rec_hdr *rec,void *data) +static int llog_cat_process_common (const struct lu_env *env, +struct llog_handle *cat_llh, +struct llog_rec_hdr *rec, +struct llog_handle **llhp) { -struct llog_process_data *d = data; -struct llog_logid_rec *lir = (struct llog_logid_rec *)rec; -struct llog_handle *llh; +struct llog_logid_rec *lir =container_of (rec,typeof (*lir), lid_hdr);int rc; -if (rec->lrh_type !=LLOG_LOGID_MAGIC ) { +if (rec->lrh_type !=le32_to_cpu (LLOG_LOGID_MAGIC )) { + rc = -EINVAL ; +CWARN ("%s: invalid record in catalog " DFID ":%x: rc = %d\n" , + cat_llh->lgh_ctxt->loc_obd->obd_name, +PFID (&cat_llh->lgh_id.lgl_oi.oi_fid), + cat_llh->lgh_id.lgl_ogen, rc); +CERROR ("invalid record in catalog\n" );return -EINVAL ; }CDEBUG (D_HA ,"processing log " DFID ":%x at index %u of catalog " DFID "\n" ,PFID (&lir->lid_id.lgl_oi.oi_fid), lir->lid_id.lgl_ogen, - rec->lrh_index,PFID (&cat_llh->lgh_id.lgl_oi.oi_fid)); +le32_to_cpu (rec->lrh_index), +PFID (&cat_llh->lgh_id.lgl_oi.oi_fid)); - rc =llog_cat_id2handle (env, cat_llh, &llh, &lir->lid_id); + rc =llog_cat_id2handle (env, cat_llh, llhp, &lir->lid_id);if (rc) { -CERROR ("%s: cannot find handle for llog " DFID ": %d\n" , - cat_llh->lgh_ctxt->loc_obd->obd_name, -PFID (&lir->lid_id.lgl_oi.oi_fid), rc); +CWARN ("%s: can't find llog handle " DFID ":%x: rc = %d\n" , + cat_llh->lgh_ctxt->loc_obd->obd_name, +PFID (&lir->lid_id.lgl_oi.oi_fid), + lir->lid_id.lgl_ogen, rc); +return rc; } +return rc; +} + +static int llog_cat_process_cb (const struct lu_env *env, +struct llog_handle *cat_llh, +struct llog_rec_hdr *rec,void *data) +{ +struct llog_process_data *d = data; +struct llog_handle *llh =NULL ; +int rc; + + rc =llog_cat_process_common (env, cat_llh, rec, &llh); +if (rc) + goto out; +if (rec->lrh_index < d->lpd_startcat)/* Skip processing of the logs until startcat */ rc =0 ;
@@ -183,6 +206,7 @@ static int llog_cat_process_cb(const struct lu_env *env, NULL, false); } +out:llog_handle_put (llh);return rc;
Signature · 1 line
1.8.3.1
PATCH 21/151 lustre: llog: consolidate common error checking
Quoted 39 lines · James Simmons
From: Andreas Dilger <[email protected]>
Consolidate error checking for corrupted and empty llog files detected when the llog file is being opened into a single routine
llog_cat_process_common().WC-bug-id: https://jira.whamcloud.com/browse/LU-9153
Lustre-commit:b522f7b4b1de("LU-9153 llog: consolidate common error checking")Signed-off-by Andreas Dilger <[email protected]>Reviewed-on: https://review.whamcloud.com/26211
Reviewed-by Faccini Bruno <[email protected]>Reviewed-by Mike Pershin <[email protected]>Reviewed-by Oleg Drokin <[email protected]>Signed-off-by James Simmons <[email protected]>--- fs/lustre/obdclass/llog_cat.c | 48 ++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 12 deletions(-)fs/lustre/obdclass/llog_cat.c +5
-6@@ -140,32 +140,55 @@ int llog_cat_close(const struct lu_env *env, struct llog_handle *cathandle) }EXPORT_SYMBOL (llog_cat_close); -static int llog_cat_process_cb (const struct lu_env *env, -struct llog_handle *cat_llh, -struct llog_rec_hdr *rec,void *data) +static int llog_cat_process_common (const struct lu_env *env, +struct llog_handle *cat_llh, +struct llog_rec_hdr *rec, +struct llog_handle **llhp) { -struct llog_process_data *d = data; -struct llog_logid_rec *lir = (struct llog_logid_rec *)rec; -struct llog_handle *llh; +struct llog_logid_rec *lir =container_of (rec,typeof (*lir), lid_hdr);int rc;
fs/lustre/obdclass/llog_cat.c int llog_cat_close(const struct lu_env *env, struct llog_handle *cathandle) · line 158
Quoted 10 lines
fs/lustre/obdclass/llog_cat.c +7
-1-if (rec->lrh_type !=LLOG_LOGID_MAGIC ) { +if (rec->lrh_type !=le32_to_cpu (LLOG_LOGID_MAGIC )) { + rc = -EINVAL ; +CWARN ("%s: invalid record in catalog " DFID ":%x: rc = %d\n" , + cat_llh->lgh_ctxt->loc_obd->obd_name, +PFID (&cat_llh->lgh_id.lgl_oi.oi_fid), + cat_llh->lgh_id.lgl_ogen, rc); +CERROR ("invalid record in catalog\n" );
You've got an extra 'invalid record in catalog' here. The original deleted this.
fs/lustre/obdclass/llog_cat.c int llog_cat_close(const struct lu_env *env, struct llog_handle *cathandle) · line 159
Quoted 1 line
fs/lustre/obdclass/llog_cat.c +0
-0return -EINVAL ;
And this is "RETURN(rc);" in the original..
Thanks,
NeilBrown
Quoted 50 lines
fs/lustre/obdclass/llog_cat.c +24
-5}CDEBUG (D_HA ,"processing log " DFID ":%x at index %u of catalog " DFID "\n" ,PFID (&lir->lid_id.lgl_oi.oi_fid), lir->lid_id.lgl_ogen, - rec->lrh_index,PFID (&cat_llh->lgh_id.lgl_oi.oi_fid)); +le32_to_cpu (rec->lrh_index), +PFID (&cat_llh->lgh_id.lgl_oi.oi_fid)); - rc =llog_cat_id2handle (env, cat_llh, &llh, &lir->lid_id); + rc =llog_cat_id2handle (env, cat_llh, llhp, &lir->lid_id);if (rc) { -CERROR ("%s: cannot find handle for llog " DFID ": %d\n" , - cat_llh->lgh_ctxt->loc_obd->obd_name, -PFID (&lir->lid_id.lgl_oi.oi_fid), rc); +CWARN ("%s: can't find llog handle " DFID ":%x: rc = %d\n" , + cat_llh->lgh_ctxt->loc_obd->obd_name, +PFID (&lir->lid_id.lgl_oi.oi_fid), + lir->lid_id.lgl_ogen, rc); +return rc; } +return rc; +} + +static int llog_cat_process_cb (const struct lu_env *env, +struct llog_handle *cat_llh, +struct llog_rec_hdr *rec,void *data) +{ +struct llog_process_data *d = data; +struct llog_handle *llh =NULL ; +int rc; + + rc =llog_cat_process_common (env, cat_llh, rec, &llh); +if (rc) + goto out; +if (rec->lrh_index < d->lpd_startcat)/* Skip processing of the logs until startcat */ rc =0 ;@@ -183,6 +206,7 @@ static int llog_cat_process_cb(const struct lu_env *env, NULL, false); } +out:llog_handle_put (llh);return rc;Signature · 1 line
1.8.3.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20191001/7c68f75e/attachment-0001.sig>
PATCH 21/151 lustre: llog: consolidate common error checking
Quoted 55 lines
Quoted 40 lines
From: Andreas Dilger <[email protected]>
Consolidate error checking for corrupted and empty llog files detected when the llog file is being opened into a single routine
llog_cat_process_common().WC-bug-id: https://jira.whamcloud.com/browse/LU-9153
Lustre-commit:b522f7b4b1de("LU-9153 llog: consolidate common error checking")Signed-off-by Andreas Dilger <[email protected]>Reviewed-on: https://review.whamcloud.com/26211
Reviewed-by Faccini Bruno <[email protected]>Reviewed-by Mike Pershin <[email protected]>Reviewed-by Oleg Drokin <[email protected]>Signed-off-by James Simmons <[email protected]>--- fs/lustre/obdclass/llog_cat.c | 48 ++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 12 deletions(-)fs/lustre/obdclass/llog_cat.c +5
-6@@ -140,32 +140,55 @@ int llog_cat_close(const struct lu_env *env, struct llog_handle *cathandle) }EXPORT_SYMBOL (llog_cat_close); -static int llog_cat_process_cb (const struct lu_env *env, -struct llog_handle *cat_llh, -struct llog_rec_hdr *rec,void *data) +static int llog_cat_process_common (const struct lu_env *env, +struct llog_handle *cat_llh, +struct llog_rec_hdr *rec, +struct llog_handle **llhp) { -struct llog_process_data *d = data; -struct llog_logid_rec *lir = (struct llog_logid_rec *)rec; -struct llog_handle *llh; +struct llog_logid_rec *lir =container_of (rec,typeof (*lir), lid_hdr);int rc;fs/lustre/obdclass/llog_cat.c int llog_cat_close(const struct lu_env *env, struct llog_handle *cathandle) · line 158
Quoted 10 lines
fs/lustre/obdclass/llog_cat.c +7
-1-if (rec->lrh_type !=LLOG_LOGID_MAGIC ) { +if (rec->lrh_type !=le32_to_cpu (LLOG_LOGID_MAGIC )) { + rc = -EINVAL ; +CWARN ("%s: invalid record in catalog " DFID ":%x: rc = %d\n" , + cat_llh->lgh_ctxt->loc_obd->obd_name, +PFID (&cat_llh->lgh_id.lgl_oi.oi_fid), + cat_llh->lgh_id.lgl_ogen, rc); +CERROR ("invalid record in catalog\n" );You've got an extra 'invalid record in catalog' here. The original deleted this.
Quoted 3 lines
fs/lustre/obdclass/llog_cat.c int llog_cat_close(const struct lu_env *env, struct llog_handle *cathandle) · line 159
Quoted 1 line
fs/lustre/obdclass/llog_cat.c +0
-0return -EINVAL ;And this is "RETURN(rc);" in the original..
Thanks for finding that.
Quoted 55 lines
Thanks,
NeilBrownQuoted 50 lines
fs/lustre/obdclass/llog_cat.c +24
-5}CDEBUG (D_HA ,"processing log " DFID ":%x at index %u of catalog " DFID "\n" ,PFID (&lir->lid_id.lgl_oi.oi_fid), lir->lid_id.lgl_ogen, - rec->lrh_index,PFID (&cat_llh->lgh_id.lgl_oi.oi_fid)); +le32_to_cpu (rec->lrh_index), +PFID (&cat_llh->lgh_id.lgl_oi.oi_fid)); - rc =llog_cat_id2handle (env, cat_llh, &llh, &lir->lid_id); + rc =llog_cat_id2handle (env, cat_llh, llhp, &lir->lid_id);if (rc) { -CERROR ("%s: cannot find handle for llog " DFID ": %d\n" , - cat_llh->lgh_ctxt->loc_obd->obd_name, -PFID (&lir->lid_id.lgl_oi.oi_fid), rc); +CWARN ("%s: can't find llog handle " DFID ":%x: rc = %d\n" , + cat_llh->lgh_ctxt->loc_obd->obd_name, +PFID (&lir->lid_id.lgl_oi.oi_fid), + lir->lid_id.lgl_ogen, rc); +return rc; } +return rc; +} + +static int llog_cat_process_cb (const struct lu_env *env, +struct llog_handle *cat_llh, +struct llog_rec_hdr *rec,void *data) +{ +struct llog_process_data *d = data; +struct llog_handle *llh =NULL ; +int rc; + + rc =llog_cat_process_common (env, cat_llh, rec, &llh); +if (rc) + goto out; +if (rec->lrh_index < d->lpd_startcat)/* Skip processing of the logs until startcat */ rc =0 ;@@ -183,6 +206,7 @@ static int llog_cat_process_cb(const struct lu_env *env, NULL, false); } +out:llog_handle_put (llh);return rc;Signature · 1 line
1.8.3.1
PATCH 22/151 lustre: llite: NULL pointer dereference in cl_object_top()
In reply to James SimmonsFrom: Andriy Skulysh <[email protected]>
During race condition iget5_locked() returns inode without I_NEW bit for the second thread. If cl_file_inode_init() failed for the first thread lli_clob isn't valid for corresponding fid.
But failed inode is marked bad, check it.
WC-bug-id: https://jira.whamcloud.com/browse/LU-6436
Seagate-bug-id: MRP-4035
Lustre-commit: 13c8d5e4bebf ("LU-6436 llite: NULL pointer dereference in cl_object_top()")
Reviewed-on: https://review.whamcloud.com/27777
--- fs/lustre/llite/namei.c | 3 +++ 1 file changed, 3 insertions(+)
fs/lustre/llite/namei.c +3 -0
@@ -133,6 +133,9 @@ struct inode *ll_iget(struct super_block *sb, ino_t hash,inode_has_no_xattr (inode);unlock_new_inode (inode); } + }else if (is_bad_inode (inode)) { +iput (inode); + inode =ERR_PTR (-ESTALE ); }else if (!(inode->i_state & (I_FREEING |I_CLEAR ))) { rc =ll_update_inode (inode, md); CDEBUG(D_VFSTRACE, "got inode: " DFID "(%p): rc = %d\n",
Signature · 1 line
1.8.3.1
PATCH 23/151 lustre: ptlrpc: remove incorrect pid printing
In reply to James SimmonsFrom: Sebastien Buisson <[email protected]>
Remove incorrect pid printing in ptlrpc_send_new_req(). It was always printing 0, and current pid is part of the message header anyway.
WC-bug-id: https://jira.whamcloud.com/browse/LU-7760
Lustre-commit: 3283170eeb05 ("LU-7760 ptlrpc: remove incorrect pid printing")
Reviewed-on: https://review.whamcloud.com/27336
--- fs/lustre/ptlrpc/client.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/lustre/ptlrpc/client.c +1 -2
@@ -1551,8 +1551,7 @@ static int ptlrpc_send_new_req(struct ptlrpc_request *req)spin_unlock (&req->rq_lock);DEBUG_REQ (D_HA , req, -"req from PID %d waiting for recovery: (%s != %s)" , -lustre_msg_get_status (req->rq_reqmsg), +"req waiting for recovery: (%s != %s)" ,ptlrpc_import_state_name (req->rq_send_state),ptlrpc_import_state_name (imp->imp_state));LASSERT (list_empty (&req->rq_list));
Signature · 1 line
1.8.3.1
PATCH 24/151 lnet: Fix lost lock
In reply to James SimmonsFrom: Dmitry Eremin <[email protected]>
Unlock lnet_net_lock in case of error in function lnet_dyn_del_net().
WC-bug-id: https://jira.whamcloud.com/browse/LU-9769
Lustre-commit: 9761b5c63103 ("LU-9769 lnet: Fix lost lock")
Reviewed-on: https://review.whamcloud.com/28039
--- net/lnet/lnet/api-ni.c | 1 + 1 file changed, 1 insertion(+)
net/lnet/lnet/api-ni.c +1 -0
@@ -2906,6 +2906,7 @@ int lnet_dyn_del_ni(struct lnet_ioctl_config_ni *conf) net =lnet_get_net_locked (net_id);if (!net) { +lnet_net_unlock (0 ); rc = -EINVAL ; goto out; }
Signature · 1 line
1.8.3.1
PATCH 25/151 lustre: llite: Reduce overhead for ll_do_fast_read
In reply to James SimmonsFrom: Patrick Farrell <[email protected]>
In ll_do_fast_read, looking up a cl_env adds some overhead, and can also cause spinlock contention on older kernels.
Fast read can safely use the preallocated percpu cl_env, so do that to reduce overhead.
SLES numbers on recent Xeon, CentOS numbers on VMs on older hardware. SLES has queued spinlocks and scales perfectly with multiple threads, with or without this patch. CentOS scales poorly at small I/O sizes without this patch.
SLES is SLES12SP2, CentOS is CentOS 7.3.
SLES:
1 thread
8b 1K 1M
Without: 23 2200 6800 With: 27.5 2500 7200
4 threads
8b 1K 1M Without: 90 8700 27000 With: 108 10000 28000
Earlier kernel (CentOS 7.3):
1 thread
8b 1K 1M Without: 9 1000 5100 with: 12 1300 5800
4 threads
8b 1K 1M Without: 22 2400 17000 With: 48 4900 20000
WC-bug-id: https://jira.whamcloud.com/browse/LU-9749
Lustre-commit: c084c6215851 ("LU-9749 llite: Reduce overhead for ll_do_fast_read")
Reviewed-on: https://review.whamcloud.com/27970
--- fs/lustre/llite/file.c | 16 ++++++---------- fs/lustre/llite/rw.c | 29 ++++++++++++++++++++--------- 2 files changed, 26 insertions(+), 19 deletions(-)
fs/lustre/llite/file.c +6 -10
@@ -1270,15 +1270,13 @@ static void ll_io_init(struct cl_io *io, const struct file *file, int write)* doesn't make the situation worse on single node but it may interleave write * results from multiple nodes due to short read handling in ll_file_aio_read(). * -* @env: lu_env * @iocb: kiocb from kernel * @iter: user space buffers where the data will be copied * * Returns: number of bytes have been read, or error code if error occurred. */ static ssize_t -ll_do_fast_read (const struct lu_env *env,struct kiocb *iocb, -struct iov_iter *iter) +ll_do_fast_read (struct kiocb *iocb,struct iov_iter *iter) {ssize_t result;
@@ -1292,9 +1290,7 @@ static void ll_io_init(struct cl_io *io, const struct file *file, int write)if (iocb->ki_filp->f_flags &O_DIRECT )return 0 ; -ll_cl_add (iocb->ki_filp, env,NULL ,LCC_RW ); result =generic_file_read_iter (iocb, iter); -ll_cl_remove (iocb->ki_filp, env);/* * If the first page is not in cache, generic_file_aio_read() will be
@@ -1319,14 +1315,14 @@ static ssize_t ll_file_read_iter(struct kiocb *iocb, struct iov_iter *to)u16 refcheck;ssize_t rc2; + result =ll_do_fast_read (iocb, to); +if (result <0 ||iov_iter_count (to) ==0 ) + goto out; + env =cl_env_get (&refcheck);if (IS_ERR (env))return PTR_ERR (env); - result =ll_do_fast_read (env, iocb, to); -if (result <0 ||iov_iter_count (to) ==0 ) - goto out; - args =ll_env_args (env); args->u.normal.via_iter = to; args->u.normal.via_iocb = iocb;
@@ -1338,8 +1334,8 @@ static ssize_t ll_file_read_iter(struct kiocb *iocb, struct iov_iter *to)else if (result ==0 ) result = rc2; -out:cl_env_put (env, &refcheck); +out:return result; }
fs/lustre/llite/rw.c +20 -9
@@ -1158,22 +1158,21 @@ int ll_readpage(struct file *file, struct page *vmpage) {struct cl_object *clob =ll_i2info (file_inode (file))->lli_clob;struct ll_cl_context *lcc; -const struct lu_env *env; -struct cl_io *io; +const struct lu_env *env =NULL ; +struct cl_io *io =NULL ;struct cl_page *page;int result; lcc =ll_cl_find (file); -if (!lcc) { -unlock_page (vmpage); -return -EIO ; +if (lcc) { + env = lcc->lcc_env; + io = lcc->lcc_io; } - env = lcc->lcc_env; - io = lcc->lcc_io;if (!io) {/* fast read */ struct ll_file_data *fd =LUSTRE_FPRIVATE (file);struct ll_readahead_state *ras = &fd->fd_ras; +struct lu_env *local_env =NULL ;struct inode *inode =file_inode (file);struct vvp_page *vpg;
@@ -1189,11 +1188,16 @@ int ll_readpage(struct file *file, struct page *vmpage)return result; } +if (!env) { + local_env =cl_env_percpu_get (); + env = local_env; + } + vpg =cl2vvp_page (cl_object_page_slice (page->cp_obj, page));if (vpg->vpg_defer_uptodate) {enum ras_update_flags flags =LL_RAS_HIT ; -if (lcc->lcc_type ==LCC_MMAP ) +if (lcc && lcc->lcc_type ==LCC_MMAP ) flags |=LL_RAS_MMAP ;/*
@@ -1220,8 +1224,15 @@ int ll_readpage(struct file *file, struct page *vmpage) } } -unlock_page (vmpage); +/* release page refcount before unlocking the page to ensure +* the object won't be destroyed in the calling path of +* cl_page_put(). Please see comment in ll_releasepage(). +*/ cl_page_put (env, page); +unlock_page (vmpage); +if (local_env) +cl_env_percpu_put (local_env); +return result; }
Signature · 1 line
1.8.3.1
PATCH 26/151 lustre: ptlrpc: change cr_sent_tv from timespec to ktime
In reply to James SimmonsInstead of calculating timediff by manually subtracting the current timespec64 from cr_sent_tv we can convert cr_sent_tv to ktime_t and simplify the code.
WC-id: https://jira.hpdd.intel.com/browse/LU-4423
Lustre-commit: de7a14b54637 ("LU-4423 ptlrpc: use 64-bit times for request times")
Reviewed-on: https://review.whamcloud.com/24977
WC-id: https://jira.hpdd.intel.com/browse/LU-9683
Lustre-commit: 61c48e79fdfb ("LU-9683 ptlrpc: fix argument misorder")
Reviewed-on: https://review.whamcloud.com/28027
--- fs/lustre/include/lustre_net.h | 6 +++--- fs/lustre/ptlrpc/client.c | 11 +++++------ fs/lustre/ptlrpc/niobuf.c | 2 +- fs/lustre/ptlrpc/service.c | 33 +++++++++++++-------------------- 4 files changed, 22 insertions(+), 30 deletions(-)
fs/lustre/include/lustre_net.h +3 -3
@@ -537,8 +537,8 @@ struct ptlrpc_cli_req {long cr_delay_limit;/** time request was first queued */ unsigned long cr_queued_time; -/** request sent timeval */ -struct timespec64 cr_sent_tv; +/** request sent in nanosseconds */ +ktime_t cr_sent_ns;/** time for request really sent out */ time64_t cr_sent_out;/** when req reply unlink must finish. */
@@ -600,7 +600,7 @@ struct ptlrpc_cli_req {#define rq_bulk rq_cli.cr_bulk#define rq_delay_limit rq_cli.cr_delay_limit#define rq_queued_time rq_cli.cr_queued_time -#define rq_sent_tv rq_cli.cr_sent_tv +#define rq_sent_ns rq_cli.cr_sent_ns#define rq_real_sent rq_cli.cr_sent_out#define rq_reply_deadline rq_cli.cr_reply_deadline#define rq_bulk_deadline rq_cli.cr_bulk_deadline
fs/lustre/ptlrpc/client.c +5 -6
@@ -1325,8 +1325,8 @@ static int after_reply(struct ptlrpc_request *req)struct obd_import *imp = req->rq_import;struct obd_device *obd = req->rq_import->imp_obd;int rc; -struct timespec64 work_start; -long timediff; +ktime_t work_start; +s64 timediff;u64 committed;LASSERT (obd);
@@ -1355,10 +1355,9 @@ static int after_reply(struct ptlrpc_request *req)return 0 ; } -ktime_get_real_ts64 (&work_start); - timediff = (work_start.tv_sec - req->rq_sent_tv.tv_sec) *USEC_PER_SEC + - (work_start.tv_nsec - req->rq_sent_tv.tv_nsec) / -NSEC_PER_USEC ; + work_start =ktime_get_real (); + timediff =ktime_us_delta (work_start, req->rq_sent_ns); +/* * NB Until this point, the whole of the incoming message, * including buflens, status etc is in the sender's byte order.
fs/lustre/ptlrpc/niobuf.c +1 -1
@@ -684,7 +684,7 @@ int ptl_send_rpc(struct ptlrpc_request *request, int noreply)OBD_FAIL_TIMEOUT (OBD_FAIL_PTLRPC_DELAY_SEND , request->rq_timeout +5 ); -ktime_get_real_ts64 (&request->rq_sent_tv); + request->rq_sent_ns =ktime_get_real (); request->rq_sent =ktime_get_real_seconds ();/* We give the server rq_timeout secs to process the req, and * add the network latency for our local timeout.
fs/lustre/ptlrpc/service.c +13 -20
@@ -1649,12 +1649,11 @@ static bool ptlrpc_server_normal_pending(struct ptlrpc_service_part *svcpt, {struct ptlrpc_service *svc = svcpt->scp_service;struct ptlrpc_request *request; -struct timespec64 work_start; -struct timespec64 work_end; -struct timespec64 timediff; -struct timespec64 arrived; -unsigned long timediff_usecs; -unsigned long arrived_usecs; +ktime_t work_start; +ktime_t work_end; +ktime_t arrived; +s64 timediff_usecs; +s64 arrived_usecs;int fail_opc =0 ; request =ptlrpc_server_request_get (svcpt, false);
@@ -1676,10 +1675,9 @@ static bool ptlrpc_server_normal_pending(struct ptlrpc_service_part *svcpt,if (OBD_FAIL_CHECK (OBD_FAIL_PTLRPC_DUMP_LOG ))libcfs_debug_dumplog (); -ktime_get_real_ts64 (&work_start); - timediff =timespec64_sub (work_start, request->rq_arrival_time); - timediff_usecs = timediff.tv_sec *USEC_PER_SEC + - timediff.tv_nsec /NSEC_PER_USEC ; + work_start =ktime_get_real (); + arrived =timespec64_to_ktime (request->rq_arrival_time); + timediff_usecs =ktime_us_delta (work_start, arrived);if (likely (svc->srv_stats)) {lprocfs_counter_add (svc->srv_stats,PTLRPC_REQWAIT_CNTR , timediff_usecs);
@@ -1746,15 +1744,11 @@ static bool ptlrpc_server_normal_pending(struct ptlrpc_service_part *svcpt, (s64)ktime_get_real_seconds() - request->rq_deadline); } -ktime_get_real_ts64 (&work_end); - timediff =timespec64_sub (work_end, work_start); - timediff_usecs = timediff.tv_sec *USEC_PER_SEC + - timediff.tv_nsec /NSEC_PER_USEC ; - arrived =timespec64_sub (work_end, request->rq_arrival_time); - arrived_usecs = arrived.tv_sec *USEC_PER_SEC + - arrived.tv_nsec /NSEC_PER_USEC ; + work_end =ktime_get_real (); + timediff_usecs =ktime_us_delta (work_end, work_start); + arrived_usecs =ktime_us_delta (work_end, arrived); CDEBUG(D_RPCTRACE, - "Handled RPC pname:cluuid+ref:pid:xid:nid:opc %s:%s+%d:%d:x%llu:%s:%d Request processed in %ldus (%ldus total) trans %llu rc %d/%d\n", + "Handled RPC pname:cluuid+ref:pid:xid:nid:opc %s:%s+%d:%d:x%llu:%s:%d Request processed in %lldus (%lldus total) trans %llu rc %d/%d\n", current->comm, (request->rq_export ? (char *)request->rq_export->exp_client_uuid.uuid : "0"),
@@ -1787,8 +1781,7 @@ static bool ptlrpc_server_normal_pending(struct ptlrpc_service_part *svcpt,DEBUG_REQ (D_ADAPTTO , request,"sent %d early replies before finishing in %llds" , request->rq_early_count, - (s64 )work_end.tv_sec - - request->rq_arrival_time.tv_sec); + arrived_usecs /USEC_PER_SEC ); }ptlrpc_server_finish_active_request (svcpt, request);
Signature · 1 line
1.8.3.1
PATCH 27/151 lustre: ptlrpc: Use C99 initializer in ptlrpc_register_rqbd()
In reply to James SimmonsFrom: Steve Guminski <[email protected]>
This patch makes no functional changes. The struct initializer in ptlrpc_register_rqbd() is updated to C99 syntax.
C89 positional initializers require values to be placed in the correct order. This will cause errors if the fields of the struct definition are reordered or fields are added or removed. C99 named initializers avoid this problem, and also automatically clear any values that are not explicitly set.
WC-bug-id: https://jira.whamcloud.com/browse/LU-6210
Lustre-commit: 3af10d510078 ("LU-6210 ptlrpc: Use C99 initializer in ptlrpc_register_rqbd()")
Reviewed-on: https://review.whamcloud.com/28479
--- fs/lustre/ptlrpc/niobuf.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
fs/lustre/ptlrpc/niobuf.c +4 -1
@@ -747,7 +747,10 @@ int ptl_send_rpc(struct ptlrpc_request *request, int noreply)int ptlrpc_register_rqbd (struct ptlrpc_request_buffer_desc *rqbd) {struct ptlrpc_service *service = rqbd->rqbd_svcpt->scp_service; -static struct lnet_process_id match_id = {LNET_NID_ANY ,LNET_PID_ANY }; +static struct lnet_process_id match_id = { + .nid =LNET_NID_ANY , + .pid =LNET_PID_ANY + };int rc;struct lnet_md md;struct lnet_handle_me me_h;
Signature · 1 line
1.8.3.1
PATCH 28/151 lustre: lmv: stripe dir page may be released mistakenly
In reply to James SimmonsFrom: Lai Siyao <[email protected]>
stripe_dirent_next() may put_stripe_dirent() while its dirent is still in use, e.g. lmv_dirent_next() popped stripe last dirent, when it can't point sd_ent to next, but it shouldn't release stripe dir page.
stripe_dirent->sd_ent should be set NULL when its dir page is released, which can avoid misuse.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9857
Lustre-commit: beadbad429ae ("LU-9857 lmv: stripe dir page may be released mistakenly")
Reviewed-on: https://review.whamcloud.com/28548
--- fs/lustre/lmv/lmv_obd.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
fs/lustre/lmv/lmv_obd.c +10 -7
@@ -2023,6 +2023,7 @@ static inline void put_stripe_dirent(struct stripe_dirent *stripe)kunmap (stripe->sd_page);put_page (stripe->sd_page); stripe->sd_page =NULL ; + stripe->sd_ent =NULL ; } }
@@ -2045,20 +2046,24 @@ static struct lu_dirent *stripe_dirent_next(struct lmv_dir_ctxt *ctxt,LASSERT (stripe == &ctxt->ldc_stripes[stripe_index]); +if (stripe->sd_eof) +return NULL ; +if (ent) { ent =lu_dirent_next (ent);if (!ent) { check_eof: end =le64_to_cpu (stripe->sd_dp->ldp_hash_end); -put_stripe_dirent (stripe); - +LASSERTF (hash <= end,"hash %llx end %llx\n" , + hash, end);if (end ==MDS_DIR_END_OFF ) { stripe->sd_ent =NULL ; stripe->sd_eof = true;return NULL ; } -LASSERT (hash <= end); + +put_stripe_dirent (stripe); hash = end; } }
@@ -2114,10 +2119,8 @@ static struct lu_dirent *stripe_dirent_next(struct lmv_dir_ctxt *ctxt, le16_to_cpu(ent->lde_namelen)) == 0))continue ; -if (le64_to_cpu (ent->lde_hash) < hash) -continue ; - -break ; +if (le64_to_cpu (ent->lde_hash) >= hash) +break ; }if (!ent)
Signature · 1 line
1.8.3.1
PATCH 29/151 lnet: selftest: Use C99 struct initializer in framework.c
In reply to James SimmonsFrom: Steve Guminski <[email protected]>
This patch makes no functional changes. The struct initializer in framework.c is updated to C99 syntax.
C89 positional initializers require values to be placed in the correct order. This will cause errors if the fields of the struct definition are reordered or fields are added or removed. C99 named initializers avoid this problem, and also automatically clear any values that are not explicitly set.
WC-bug-id: https://jira.whamcloud.com/browse/LU-6210
Lustre-commit: 60c9a2d351d4 ("LU-6210 lnet: Use C99 struct initializer in framework.c")
Reviewed-on: https://review.whamcloud.com/28436
--- net/lnet/selftest/framework.c | 42 +++++++----------------------------------- 1 file changed, 7 insertions(+), 35 deletions(-)
net/lnet/selftest/framework.c +7 -35
@@ -1584,41 +1584,13 @@ struct srpc_client_rpc * }static struct srpc_service sfw_services[] = { - { -/* sv_id */ SRPC_SERVICE_DEBUG , -/* sv_name */ "debug" , -0 - }, - { -/* sv_id */ SRPC_SERVICE_QUERY_STAT , -/* sv_name */ "query stats" , -0 - }, - { -/* sv_id */ SRPC_SERVICE_MAKE_SESSION , -/* sv_name */ "make session" , -0 - }, - { -/* sv_id */ SRPC_SERVICE_REMOVE_SESSION , -/* sv_name */ "remove session" , -0 - }, - { -/* sv_id */ SRPC_SERVICE_BATCH , -/* sv_name */ "batch service" , -0 - }, - { -/* sv_id */ SRPC_SERVICE_TEST , -/* sv_name */ "test service" , -0 - }, - { -/* sv_id */ 0 , -/* sv_name */ NULL , -0 - } + { .sv_id =SRPC_SERVICE_DEBUG , .sv_name ="debug" , }, + { .sv_id =SRPC_SERVICE_QUERY_STAT , .sv_name ="query stats" , }, + { .sv_id =SRPC_SERVICE_MAKE_SESSION , .sv_name ="make session" , }, + { .sv_id =SRPC_SERVICE_REMOVE_SESSION , .sv_name ="remove session" , }, + { .sv_id =SRPC_SERVICE_BATCH , .sv_name ="batch service" , }, + { .sv_id =SRPC_SERVICE_TEST , .sv_name ="test service" , }, + { .sv_id =0 , } };int
Signature · 1 line
1.8.3.1
PATCH 30/151 lnet: fix memory leak and lnet_interfaces_max
In reply to James SimmonsFrom: Amir Shehata <[email protected]>
Free buffer allocated for discover command.
Set lnet_interfaces_max to LNET_INTERFACES_MAX_DEFAULT if it's not defined or if it's being set to something below LNET_INTERFACES_MIN.
For lnet_ping() and lnet_discover() if the provided space can fit more NIDs than lnet_interfaces_max then ensure only lnet_interfaces_max is copied into the buffer.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9909
Lustre-commit: 81d4f7a25319 ("LU-9909 lnet: fix memory leak and lnet_interfaces_max")
Reviewed-on: https://review.whamcloud.com/28702
--- net/lnet/lnet/api-ni.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
net/lnet/lnet/api-ni.c +16 -5
@@ -170,8 +170,8 @@ static int lnet_discover(struct lnet_process_id id, u32 force,if (value <LNET_INTERFACES_MIN ) {CWARN ("max interfaces provided are too small, setting to %d\n" , -LNET_INTERFACES_MIN ); - value =LNET_INTERFACES_MIN ; +LNET_INTERFACES_MAX_DEFAULT ); + value =LNET_INTERFACES_MAX_DEFAULT ; } *(int *)kp->arg = value;
@@ -3407,9 +3407,15 @@ static int lnet_ping(struct lnet_process_id id, signed long timeout,int rc2;/* n_ids limit is arbitrary */ -if (n_ids <=0 || n_ids > lnet_interfaces_max || id.nid ==LNET_NID_ANY ) +if (n_ids <=0 || id.nid ==LNET_NID_ANY )return -EINVAL ; +/* if the user buffer has more space than the lnet_interfaces_max +* then only fill it up to lnet_interfaces_max +*/ +if (n_ids > lnet_interfaces_max) + n_ids = lnet_interfaces_max; +if (id.pid ==LNET_PID_ANY ) id.pid =LNET_PID_LUSTRE ;
@@ -3575,13 +3581,18 @@ static int lnet_ping(struct lnet_process_id id, signed long timeout,int max_intf = lnet_interfaces_max;if (n_ids <=0 || - id.nid ==LNET_NID_ANY || - n_ids > max_intf) + id.nid ==LNET_NID_ANY )return -EINVAL ;if (id.pid ==LNET_PID_ANY ) id.pid =LNET_PID_LUSTRE ; +/* if the user buffer has more space than the max_intf +* then only fill it up to max_intf +*/ +if (n_ids > max_intf) + n_ids = max_intf; + buf =kcalloc (n_ids,sizeof (*buf),GFP_KERNEL );if (!buf)return -ENOMEM ;
Signature · 1 line
1.8.3.1
PATCH 31/151 lnet: decref on peer after use
In reply to James SimmonsFrom: Amir Shehata <[email protected]>
After looking up the peer for both ping and discover we need to decref the peer so we don't lose a reference on it. This needs to be done while the mutex_lock is held to ensure the peer list remains stable.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9918
Lustre-commit: 440c1c03fe08 ("LU-9918 lnet: decref on peer after use")
Reviewed-on: https://review.whamcloud.com/28722
--- net/lnet/lnet/api-ni.c | 8 ++++++++ 1 file changed, 8 insertions(+)
net/lnet/lnet/api-ni.c +8 -0
@@ -3297,11 +3297,15 @@ u32 lnet_get_dlc_seq_locked(void)if (rc <0 )return rc; +mutex_lock (&the_lnet.ln_api_mutex); lp =lnet_find_peer (ping->ping_id.nid);if (lp) { ping->ping_id.nid = lp->lp_primary_nid; ping->mr_info =lnet_peer_is_multi_rail (lp); +lnet_peer_decref_locked (lp); } +mutex_unlock (&the_lnet.ln_api_mutex); + ping->ping_count = rc;return 0 ; }
@@ -3315,11 +3319,15 @@ u32 lnet_get_dlc_seq_locked(void) discover->ping_count);if (rc <0 )return rc; + +mutex_lock (&the_lnet.ln_api_mutex); lp =lnet_find_peer (discover->ping_id.nid);if (lp) { discover->ping_id.nid = lp->lp_primary_nid; discover->mr_info =lnet_peer_is_multi_rail (lp); +lnet_peer_decref_locked (lp); } +mutex_unlock (&the_lnet.ln_api_mutex); discover->ping_count = rc;return 0 ;
Signature · 1 line
1.8.3.1
PATCH 32/151 lnet: rediscover peer if it changed
In reply to James SimmonsFrom: Amir Shehata <[email protected]>
If the peer has changed after we unlocked the cpt then we'll need to discover the new peer.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9917
Lustre-commit: 1fc4ed3ac40a ("LU-9917 lnet: rediscover peer if it changed")
Reviewed-on: https://review.whamcloud.com/28772
--- net/lnet/lnet/peer.c | 7 +++++++ 1 file changed, 7 insertions(+)
net/lnet/lnet/peer.c +7 -0
@@ -2034,6 +2034,13 @@ static void lnet_peer_clear_discovery_error(struct lnet_peer *lp)lnet_net_unlock (LNET_LOCK_EX );lnet_net_lock (cpt); +/* If the peer has changed after we've discovered the older peer, +* then we need to discovery the new peer to make sure the +* interface information is up to date +*/ +if (lp != lpni->lpni_peer_net->lpn_peer) + goto again; +if (signal_pending (current)) rc = -EINTR ;else if (the_lnet.ln_dc_state !=LNET_DC_STATE_RUNNING )
Signature · 1 line
1.8.3.1
PATCH 33/151 lnet: resolve unsafe list access
In reply to James SimmonsFrom: Amir Shehata <[email protected]>
Use list_for_each_entry_safe() when accessing messages on pending queue. Remove the message from the list before calling lnet_finalize() or lnet_send().
When destroying the peer make sure to queue all pending messages on a global list. We can not resend them at this point because the cpt lock is held. Unlocking the cpt lock could lead to an inconsistent state. Use the discovery thread to check if the global list is not empty and if so resend all messages on the list. Use a new spin lock to protect the resend message list. I steered clear from reusing an existing lock because LNet locking is complex and reusing a lock will add to this complexity. Using a new lock makes the code easier to understand.
Verify that all lists are empty before destroying the peer_ni
WC-bug-id: https://jira.whamcloud.com/browse/LU-9921
Lustre-commit: 62c3c8d14856 ("LU-9921 lnet: resolve unsafe list access")
Reviewed-on: https://review.whamcloud.com/28723
--- include/linux/lnet/lib-types.h | 4 +++ net/lnet/lnet/api-ni.c | 15 ++++++++++ net/lnet/lnet/peer.c | 68 ++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 85 insertions(+), 2 deletions(-)
include/linux/lnet/lib-types.h +4 -0
@@ -945,6 +945,10 @@ struct lnet {struct list_head ln_net_zombie;/* the loopback NI */ struct lnet_ni *ln_loni; +/* resend messages list */ +struct list_head ln_msg_resend; +/* spin lock to protect the msg resend list */ +spinlock_t ln_msg_resend_lock;/* remote networks with routes to them */ struct list_head *ln_remote_nets_hash;
net/lnet/lnet/api-ni.c +15 -0
@@ -211,6 +211,7 @@ static int lnet_discover(struct lnet_process_id id, u32 force,lnet_init_locks (void) {spin_lock_init (&the_lnet.ln_eq_wait_lock); +spin_lock_init (&the_lnet.ln_msg_resend_lock);init_waitqueue_head (&the_lnet.ln_eq_waitq);init_waitqueue_head (&the_lnet.ln_rc_waitq);mutex_init (&the_lnet.ln_lnd_mutex);
@@ -1652,6 +1653,10 @@ static void lnet_push_target_fini(void)lnet_shutdown_lndnets (void) {struct lnet_net *net; +struct list_head resend; +struct lnet_msg *msg, *tmp; + +INIT_LIST_HEAD (&resend);/* NB called holding the global mutex */
@@ -1682,6 +1687,15 @@ static void lnet_push_target_fini(void) net_list)) != NULL)lnet_shutdown_lndnet (net); +spin_lock (&the_lnet.ln_msg_resend_lock); +list_splice (&the_lnet.ln_msg_resend, &resend); +spin_unlock (&the_lnet.ln_msg_resend_lock); + +list_for_each_entry_safe (msg, tmp, &resend, msg_list) { +list_del_init (&msg->msg_list); +lnet_finalize (msg, -ECANCELED ); + } +lnet_net_lock (LNET_LOCK_EX ); the_lnet.ln_state =LNET_STATE_SHUTDOWN ;lnet_net_unlock (LNET_LOCK_EX );
@@ -2025,6 +2039,7 @@ int lnet_lib_init(void)INIT_LIST_HEAD (&the_lnet.ln_lnds);INIT_LIST_HEAD (&the_lnet.ln_net_zombie);INIT_LIST_HEAD (&the_lnet.ln_rcd_zombie); +INIT_LIST_HEAD (&the_lnet.ln_msg_resend);INIT_LIST_HEAD (&the_lnet.ln_rcd_deathrow);/*
net/lnet/lnet/peer.c +66 -2
110 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 34/151 lustre: llite: Implement ladvise lockahead
In reply to James SimmonsFrom: Patrick Farrell <[email protected]>
Ladvise lockahead is a new feature allowing userspace to request extent locks in advance of the IO which will use them. These locks are not expanded beyond the size requested by userspace. They are intended to make it possible to address lock contention between multiple clients resulting from lock expansion. They should allow optimizing various IO patterns, notably strided writing. (Further information in LU-6179)
Asynchronous glimpse locks are a speculative version of glimpse locks, and already implement the required behavior. Lockahead requests share this behavior.
Additionally, lockahead creates extent locks in advance of IO, and so breaks the assumption that the holder of the highest lock knows the current file size.
So we also modify the ofd_intent_policy code to glimpse PW locks until it finds one it knows to be in use, taking care to send only one glimpse to each client.
The current patch allows asynchronous non-blocking lock ahead requests and synchronous blocking requests. We cannot do asynchronous blocking requests, because of deadlocks that occur in having ptlrpcd threads handle blocking lock requests.
Finally, this patch also adds another advice to disable lock expansion, setting a per-file descriptor flag. This allows user space to control whether or not lock requests on this file descriptor will undergo lock expansion.
This means if lockahead locks are not created ahead of IO (due to inherent raciness) or are cancelled by a competing IO request, the IO requests that should have used the manually requested locks will not result in expanded locks. This avoids lock ping-pong, and because the resulting locks will not extend to the end of the file, future lockahead requests can be granted. Effectively, this means that if lockahead usage for strided IO is interrupted by a competing request, it can re-assert itself.
lockahead is implented via the ladvise interface from userspace. As lockahead results in a DLM lock request rather than file advice, we do not use the lower levels of the ladvise implementation.
Note this patch has one oddity:
Cray released an earlier version of lockahead without FL_SPECULATIVE support. That version uses OBD_CONNECT_LOCKAHEAD_OLD, this new one uses OBD_CONNECT_LOCKAHEAD.
The client code in this patch is interoperable with that version, so it also advertises OBD_CONNECT_LOCKAHEAD_OLD support, but the server version is not, so the server advertises only OBD_CONNECT_LOCKAHEAD support.
Client support for the original lockahead is slated for removal after the release of 2.12. This is enforced with a compile time version test that will remove support.
WC-bug-id: https://jira.whamcloud.com/browse/LU-6179
Lustre-commit: a8dcf372f430 ("LU-6179 llite: Implement ladvise lockahead")
Reviewed-on: https://review.whamcloud.com/13564
--- Documentation/lustre/ladvise_lockahead.txt | 304 +++++++++++++++++++++++++++++ fs/lustre/include/cl_object.h | 36 +++- fs/lustre/include/lustre_dlm.h | 4 +- fs/lustre/include/lustre_dlm_flags.h | 41 +++- fs/lustre/include/lustre_export.h | 20 ++ fs/lustre/include/lustre_osc.h | 11 +- fs/lustre/ldlm/ldlm_lib.c | 1 + fs/lustre/ldlm/ldlm_lock.c | 7 +- fs/lustre/llite/file.c | 277 ++++++++++++++++++++++++-- fs/lustre/llite/glimpse.c | 27 ++- fs/lustre/llite/llite_internal.h | 9 + fs/lustre/llite/llite_lib.c | 23 ++- fs/lustre/llite/vvp_io.c | 3 + fs/lustre/lov/lov_io.c | 1 + fs/lustre/obdclass/cl_lock.c | 2 +- fs/lustre/obdclass/lprocfs_status.c | 3 +- fs/lustre/osc/osc_internal.h | 3 +- fs/lustre/osc/osc_lock.c | 71 +++++-- fs/lustre/osc/osc_request.c | 50 +++-- fs/lustre/ptlrpc/wiretest.c | 8 +- include/uapi/linux/lustre/lustre_idl.h | 12 +- include/uapi/linux/lustre/lustre_user.h | 40 +++- 22 files changed, 857 insertions(+), 96 deletions(-) create mode 100644 Documentation/lustre/ladvise_lockahead.txt
Documentation/lustre/ladvise_lockahead.txt new file +304 -0
@@ -0,0 +1,304 @@ +Ladvise Lock Ahead design + +Lock ahead is a new Lustre feature aimed at solving a long standing problem +with shared file write performance in Lustre. It requires client and server +support. It will be used primarily via the MPI-I/O library, not directly from +user applications. + +The first part of this document (sections 1 and 2) is an overview of the +problem and high level description of the solution. Section 3 explains how the +library will make use of this feature, and sections 4 and 5 describe the design +of the Lustre changes. + +1. Overview: Purpose & Interface +Lock ahead is intended to allow optimization of certain I/O patterns which +would otherwise suffer LDLM* lock contention. It allows applications to +manually request locks on specific extents of a file, avoiding the usual +server side optimizations. This applications which know their I/O pattern to +use that information to avoid false conflicts due to server side optimizations. + +*Lustre distributed lock manager. This is the locking layer shared between +clients and servers, to manage access between clients. + +Normally, clients get locks automatically as the first step of an I/O. +The client asks for a lock which covers exactly the area of interest (ie, a +read or write lock of n bytes at offset x), but the server attempts to optimize +this by expanding the lock to cover as much of the file as possible. This is +useful for a single client, but can be trouble for multiple clients. + +In cases where multiple clients wish to write to the same file, this +optimization can result in locks that conflict when the actual I/O operations +do not. This requires clients to wait for one another to complete I/O, even +when there is no conflict between actual I/O requests. This can significantly +reduce performance (Anywhere from 40-90%, depending on system specs) for some +workloads. + +The lockahead feature makes it possible to avoid this problem by acquiring the +necessary locks in advance, by explicit requests with server side extent +changes disabled. We add a new lfs advice type, LU_LADVISE_LOCKAHEAD, +which allows lock requests from userspace on the client, specifying the extent +and the I/O mode (read/write) for the lock. These lock requests explicitly +disable server side changes to the lock extent, so the lock returned to the +client covers only the extent requested. + +When using this feature, clients which intend to write to a file can request +locks to cover their I/O pattern, wait a moment for the locks to be granted, +then write or read the file. + +In this way, a set of clients which knows their I/O pattern in advance can +force the LDLM layer to grant locks appropriate for that I/O pattern. This +allows applications which are poorly handled by the default lock optimization +behavior to significantly improve their performance. + +2. I/O Pattern & Locking problems +2. A. Strided writing and MPI-I/O +There is a thorough explanation and overview of strided writing and the +benefits of this functionality in the slides from the lock ahead presentation +at LUG 2015. It is highly recommended to read that first, as the graphics are +much clearer than the prose here. + +See slides 1-13: +http://wiki.lustre.org/images/f/f9/Shared-File-Performance-in-Lustre_Farrell.pdf + +MPI-I/O uses strided writing when doing I/O from a large job to a single file. +I/O is aggregated from all the nodes running a particular application to a +small number of I/O aggregator nodes which then write out the data, in a +strided manner. + +In strided writing, different clients take turns writing different blocks of a +file (A block is some arbitrary number of bytes). Client 1 is responsible for +writes to block 0, block 2, block 4, etc., client 2 is responsible for block 1, +block 3, etc. + +Without the ability to manually request locks, strided writing is set up in +concert with Lustre file striping so each client writes to one OST. (IE, for a +file striped to three OSTs, we would write from three clients.) + +The particular case of interest is when we want to use more than one client +per OST. This is important, because an OST typically has much more bandwidth +than one client. Strided writes are non-overlapping, so they should be able to +proceed in parallel with more than one client per OST. In practice, on Lustre, +they do not, due to lock expansion. + +2. B. Locking problems +We will now describe locking when there is more than one client per OST. This +behavior is the same on a per OST basis in a file striped across multiple OSTs. +When the first client asks to write block 0, it asks for the required lock from +the server. When it receives this request, the server sees that there are no +other locks on the file. Since it assumes the client will want to write to the +file again, the server expands the lock as far as possible. In this case, it +expands the lock to the maximum file size (effectively, to infinity), then +grants it to client 1. + +When client 2 wants to write block 1, it conflicts with the expanded lock +granted to client 1. The server then must revoke (In Lustre terms, +'call back') the lock granted to client 1 so it can grant a lock to client 2. +After the lock granted to client is revoked, there are no locks on the file. +The server sees this when processing the lock request from client 2, and +expands that lock to cover the whole file. + +Client 1 then wishes to write block 3 of the file... And the cycle continues. +The two clients exchange the extended lock throughout the write, allowing only +one client to write at a time, plus latency to exchange the lock. The effect is +dramatic: Two clients are actually slower than one. (Similar behavior is seen +with more than two clients.) + +The solution is to use this new advice type to acquire locks before they are +needed. In effect, before it starts writing to the file, client 1 requests +locks on block 0, block 2, etc. It locks 'ahead' a certain (tunable) number of +locks. Client 2 does the same. Then they both begin to write, and are able to +do so in parallel. A description of the actual library implementation follows. + +3. Library implementation +Actually implementing this in the library carries a number of wrinkles. +The basic pattern is this: +Before writing, an I/O aggregator requests a certain number of locks on blocks +that it is responsible for. It may or may not ever write to these blocks, but +it takes locks knowing it might. It then begins to write, tracking how many of +the locks it has used. When the number of locks 'ahead' of the I/O is low +enough, it requests more locks in advance of the I/O. + +For technical reasons which are explained in the implementation section, these +lock requests are either asynchronous and non-blocking or synchronous and +blocking. In Lustre terms, non-blocking means if there is already a lock on +the relevant extent of the file, the manual lock request is not granted. This +means that if there is already a lock on the file (quite common; imagine +writing to a file which was previously read by another process), these lock +requests will be denied. However, once the first 'real' write arrives that +was hoping to use a lockahead lock, that write will cause the blocking lock to +be cancelled, so this interference is not fatal. + +It is of course possible for another process to get in the way by immediately +asking for a lock on the file. This is something users should try to avoid. +When writing out a file, repeatedly trying to read it will impact performance +even without this feature. + +These interfering locks can also happen if a manually requested lock is, for +some reason, not available in time for the write which intended to use it. +The lock which results from this write request is expanded using the +normal rules. So it's possible for that lock (depending on the position of +other locks at the time) to be extended to cover the rest of the file. That +will block future lockahead locks. + +The expanded lock will be revoked when a write happens (from another client) +in the range covered by that lock, but the lock for that write will be expanded +as well - And then we return to handing the lock back and forth between +clients. These expanded locks will still block future lockahead locks, +rendering them useless. + +The way to avoid this is to turn off lock expansion for I/Os which are +supposed to be using these manually requested locks. That way, if the +manually requested lock is not available, the lock request for the I/O will not +be expanded. Instead, that request (which is blocking, unlike a lockahead +request) will cancel any interfering locks, but the resulting lock will not be +expanded. This leaves the later parts of the file open, allowing future +manual lock requests to succeed. This means that if an interfering lock blocks +some manual requests, those are lost, but the next set of manual requests can +proceed as normal. + +In effect, the 'locking ahead of I/O' is interrupted, but then is able to +re-assert itself. The feature used here is referred to as 'no expansion' +locking (as only the extent required by the actual I/O operation is locked) +and is turned on with another new ladvise advice, LU_LADVISE_NOEXPAND. This +feature is added as part of the lockahead patch. The strided writing library +will use this advice on the file descriptor it uses for writing. + +4. Client side design +4. A. Ladvise lockahead +Requestlock uses the existing asynchronous lock request functionality +implemented for asynchronous glimpse locks (AGLs), a long standing Lustre +feature. AGLs are locks which are requested by statahead, which are used to +get file size information before it's requested. The key thing about an +asynchronous lock request is that it does not have a specific I/O operation +waiting for the lock. + +This means two key things: + +1. There is no OSC lock (lock layer above LDLM for data locking) associated +with the LDLM lock +2. There is no thread waiting for the LDLM lock, so lock grant processing +must be handled by the ptlrpc daemon thread which received the reply + +Since both of these issues are addressed by the asynchronous lock request code +which lockahead shares with AGL, we will not explore them in depth here. + +Finally, lockahead requests set the CEF_LOCK_NO_EXPAND flag, which tells the +OSC (the per OST layer of the client) to set LDLM_FL_NO_EXPANSION on any lock +requests. LDLM_FL_NO_EXPANSION is a new LDLM lock flag which tells the server +not to expand the lock extent. + +This leaves the user facing interface. Requestlock is implemented as a new +ladvise advice, and it uses the ladvise feature of multiple advices in one API +call to put many lock requests in to an array of advices. + +The arguments required for this advice are a mode (read or write), range (start +and end), and flags. + +The client will then make lock requests on these extents, one at a time. +Because the lock requests are asynchronous (replies are handled by ptlrpcd), +many requests can be made quickly by overlapping them, rather than waiting for +each one to complete. (This requires that they be non-blocking, as the +ptlrpcd threads must not wait in the ldlm layer.) + +4. B. LU_LADVISE_LOCKNOEXPAND +The lock no expand ladvise advice sets a boolean in a Lustre data structure +associated with a file descriptor. When an I/O is done to this file +descriptor, the flag is picked up and passed through to the ldlm layer, where +it sets LDLM_FL_NO_EXPANSION on lock requests made for that I/O. + +5. Server side changes +Implementing lockahead requires server support for LDLM_FL_NO_EXPANSION, but +it also required an additional pair of server side changes to fix issues which +came up because of lockahead. These changes are not part of the core design +instead, they are separate fixes which are required for it to work. + +5. A. Support LDLM_FL_NO_EXPANSION + +Disabling server side lock expansion is done with a new LDLM flag. This is +done with a simple check for that flag on the server before attempting to +expand the lock. If the flag is found, lock expansion is skipped. + +5. B. Implement LDLM_FL_SPECULATIVE + +As described above, lock ahead locks are non-blocking. The BLOCK_NOWAIT LDLM +flag is used now to implement some nonblocking behavior, but it only considers +group locks blocking. But, for asynchronous lock requests to work correctly, +they cannot wait for any other locks. For this purpose, we add +LDLM_FL_SPECULATIVE. This new flag is used for asynchronous lock requests, +and implements the broader non-blocking behavior they require. + +5. C. File size & ofd_intent_policy changes + +Knowing the current file size during writes is tricky on a distributed file +system, because multiple clients can be writing to a file at any time. When +writes are in progress, the server must identify which client is currently +responsible for growing the file size, and ask that client what the file size +is. + +To do this, the server uses glimpse locking (in ofd_intent_policy) to get the +current file size from the clients. This code uses the assumption that the +holder of the highest write lock (PW lock) knows the current file size. A +client learns the (then current) file size when a lock is granted. Because +only the holder of the highest lock can grow a file, either the size hasn't +changed, or that client knows the new size; so the server only has to contact +the client which holds this lock, and it knows the current file size. + +Note that the above is actually racy. When the server asks, the client can +still be writing, or another client could acquire a higher lock during this +time. The goal is a good approximation while the file is being written, and a +correct answer once all the clients are done writing. This is achieved because +once writes to a file are complete, the holder of that highest lock is +guaranteed to know the current file size. This is where manually requested +locks cause trouble. + +By creating write locks in advance of an actual I/O, lockahead breaks the +assumption that the holder of the highest lock knows the file size. + +This assumption is normally true because locks which are created as part of +IO - rather than in advance of it - are guaranteed to be 'active', IE, +involved in IO, and the holder of the highest 'active' lock always knows the +current file size, because the size is either not changing or the holder of +that lock is responsible for updating it. + +Consider: Two clients, A and B, strided writing. Each client requests, for +example, 2 manually requested locks. (Real numbers are much higher.) Client A +holds locks on segments 0 and 2, client B holds locks on segments 1 and 3. + +The request comes to write 3 segments of data. Client A writes to segment 0, +client B writes to segment 1, and client A also writes to segment 2. No data +is written to segment 3. At this point, the server checks the file size, by +glimpsing the highest lock . The lock on segment 3. Client B does not know +about the writing done by client A to segment 2, so it gives an incorrect file +size. + +This would be OK if client B had pending writes to segment 3, but it does not. +In this situation, the server will never get the correct file size while this +lock exists. + +The solution is relatively straightforward: The server needs to glimpse every +client holding a write lock (starting from the top) until we find one holding +an 'active' lock (because the size is known to be at least the size returned +from an 'active' lock), and take the largest size returned. This avoids asking +only a client which may not know the correct file size. + +Unfortunately, there is no way to know if a manually requested lock is active +from the server side. So when we see such a lock, we must send a glimpse to +the holder (unless we have already sent a glimpse to that client*). However, +because locks without LDLM_FL_NO_EXPANSION set are guaranteed to be 'active', +once we reach the first such lock, we can stop glimpsing. + +*This is because when we glimpse a specific lock, the client holding it returns +its best idea of the size information, so we only need to send one glimpse to +each client. + +This is less efficient than the standard "glimpse only the top lock" +methodology, but since we only need to glimpse one lock per client (and the +number of clients writing to the part of a file on a given OST is fairly +limited), the cost is restrained. + +Additionally, lock cancellation methods such as early lock cancel aggressively +clean up older locks, particularly when the LRU limit is exceeded, so the +total lock count should also remain manageable. + +In the end, the final verdict here is performance. Requestlock testing for the +strided I/O case has shown good performance results.
fs/lustre/include/cl_object.h +27 -9
@@ -1597,10 +1597,14 @@ enum cl_enq_flags {*/ CEF_NONBLOCK =0x00000001 ,/** -* take lock asynchronously (out of order), as it cannot -* deadlock. This is for LDLM_FL_HAS_INTENT locks used for glimpsing. +* Tell lower layers this is a glimpse request, translated to +* LDLM_FL_HAS_INTENT at LDLM layer. +* +* Also, because glimpse locks never block other locks, we count this +* as automatically compatible with other osc locks. +* (see osc_lock_compatible) */ - CEF_ASYNC = 0x00000002, + CEF_GLIMPSE = 0x00000002,/** * tell the server to instruct (though a flag in the blocking ast) an * owner of the conflicting lock, that it can drop dirty pages
@@ -1609,8 +1613,9 @@ enum cl_enq_flags {CEF_DISCARD_DATA =0x00000004 ,/** * tell the sub layers that it must be a `real' lock. This is used for -* mmapped-buffer locks and glimpse locks that must be never converted -* into lockless mode. +* mmapped-buffer locks and glimpse locks, manually requested locks +* (LU_LADVISE_LOCKAHEAD) that must be never converted into lockless +* mode. * * \see vvp_mmap_locks(), cl_glimpse_lock(). */
@@ -1627,9 +1632,16 @@ enum cl_enq_flags {*/ CEF_NEVER =0x00000010 ,/** -* for async glimpse lock. +* tell the dlm layer this is a speculative lock request +* speculative lock requests are locks which are not requested as part +* of an I/O operation. Instead, they are requested because we expect +* to use them in the future. They are requested asynchronously at the +* ptlrpc layer. +* +* Currently used for asynchronous glimpse locks and manually requested +* locks (LU_LADVISE_LOCKAHEAD). */ - CEF_AGL = 0x00000020, + CEF_SPECULATIVE = 0x00000020,/** * enqueue a lock to test DLM lock existence. */
@@ -1640,9 +1652,13 @@ enum cl_enq_flags {*/ CEF_LOCK_MATCH =BIT (7 ),/** +* tell the DLM layer to lock only the requested range +*/ +CEF_LOCK_NO_EXPAND =BIT (8 ), +/** * mask of enq_flags. */ - CEF_MASK = 0x000000ff, + CEF_MASK = 0x000001ff, };/**
@@ -1849,7 +1865,9 @@ struct cl_io {/** * O_NOATIME */ - ci_noatime:1 ; + ci_noatime:1 , +/* Tell sublayers not to expand LDLM locks requested for this IO */ + ci_lock_no_expand:1;/** * Number of pages owned by this IO. For invariant checking. */
fs/lustre/include/lustre_dlm.h +2 -2
@@ -508,8 +508,8 @@ struct ldlm_glimpse_work {*/ }; -/** The ldlm_glimpse_work is allocated on the stack and should not be freed. */ -#define LDLM_GL_WORK_NOFREE 0x1 +/* The ldlm_glimpse_work was slab allocated & must be freed accordingly.*/ +#define LDLM_GL_WORK_SLAB_ALLOCATED 0x1/** * Interval tree for extent locks.
fs/lustre/include/lustre_dlm_flags.h +36 -5
61 lines not on this page · raw message
fs/lustre/include/lustre_export.h +20 -0
@@ -149,6 +149,16 @@ static inline u64 exp_connect_flags(struct obd_export *exp)return *exp_connect_flags_ptr (exp); } +static inline u64 *exp_connect_flags2_ptr (struct obd_export *exp) +{ +return &exp->exp_connect_data.ocd_connect_flags2; +} + +static inline u64 exp_connect_flags2 (struct obd_export *exp) +{ +return *exp_connect_flags2_ptr (exp); +} +static inline int exp_max_brw_size (struct obd_export *exp) {if (exp_connect_flags (exp) &OBD_CONNECT_BRW_SIZE )
@@ -235,6 +245,16 @@ static inline bool imp_connect_disp_stripe(struct obd_import *imp)return ocd->ocd_connect_flags &OBD_CONNECT_DISP_STRIPE ; } +static inline int exp_connect_lockahead_old (struct obd_export *exp) +{ +return !!(exp_connect_flags (exp) &OBD_CONNECT_LOCKAHEAD_OLD ); +} + +static inline int exp_connect_lockahead (struct obd_export *exp) +{ +return !!(exp_connect_flags2 (exp) &OBD_CONNECT2_LOCKAHEAD ); +} +struct obd_export *class_conn2export (struct lustre_handle *conn);#define KKUC_CT_DATA_MAGIC 0x092013cea
fs/lustre/include/lustre_osc.h +10 -1
@@ -380,7 +380,16 @@ struct osc_lock {/* * For async glimpse lock. */ - ols_agl:1 ; + ols_agl:1 , +/* +* for speculative locks - asynchronous glimpse locks and ladvise +* lockahead manual lock requests +* +* Used to tell osc layer to not wait for the ldlm reply from the +* server, so the osc lock will be short lived - It only exists to +* create the ldlm request and is not updated on request completion. +*/ + ols_speculative:1; };/*
fs/lustre/ldlm/ldlm_lib.c +1 -0
@@ -558,6 +558,7 @@ int client_connect_import(const struct lu_env *env, ocd->ocd_connect_flags,"old %#llx, new %#llx\n" , data->ocd_connect_flags, ocd->ocd_connect_flags); data->ocd_connect_flags = ocd->ocd_connect_flags; + data->ocd_connect_flags2 = ocd->ocd_connect_flags2; }ptlrpc_pinger_add_import (imp);
fs/lustre/ldlm/ldlm_lock.c +6 -1
@@ -43,6 +43,8 @@#include <obd_class.h> #include "ldlm_internal.h" +struct kmem_cache *ldlm_glimpse_work_kmem; +/* lock types */ char *ldlm_lockname[] = { [0 ] ="--" ,
@@ -1756,8 +1758,11 @@ static int ldlm_work_gl_ast_lock(struct ptlrpc_request_set *rqset, void *opaq)LDLM_LOCK_RELEASE (lock); -if ((gl_work->gl_flags &LDLM_GL_WORK_NOFREE ) ==0 ) +if (gl_work->gl_flags &LDLM_GL_WORK_SLAB_ALLOCATED ) +kmem_cache_free (ldlm_glimpse_work_kmem, gl_work); +else kfree (gl_work); + gl_work =NULL ;return rc; }
fs/lustre/llite/file.c +257 -20
339 lines not on this page · raw message
fs/lustre/llite/glimpse.c +18 -9
@@ -88,7 +88,7 @@ int cl_glimpse_lock(const struct lu_env *env, struct cl_io *io,CDEBUG (D_DLMTRACE ,"Glimpsing inode " DFID "\n" ,PFID (fid));/* NOTE: this looks like DLM lock request, but it may -* not be one. Due to CEF_ASYNC flag (translated +* not be one. Due to CEF_GLIMPSE flag (translated * to LDLM_FL_HAS_INTENT by osc), this is * glimpse request, that won't revoke any * conflicting DLM locks held. Instead,
@@ -104,14 +104,10 @@ int cl_glimpse_lock(const struct lu_env *env, struct cl_io *io, *descr = whole_file; descr->cld_obj = clob; descr->cld_mode =CLM_READ ; - descr->cld_enq_flags =CEF_ASYNC |CEF_MUST ; + descr->cld_enq_flags =CEF_GLIMPSE |CEF_MUST ;if (agl) - descr->cld_enq_flags |=CEF_AGL ; + descr->cld_enq_flags |=CEF_SPECULATIVE |CEF_NONBLOCK ;/* -* CEF_ASYNC is used because glimpse sub-locks cannot -* deadlock (because they never conflict with other -* locks) and, hence, can be enqueued out-of-order. -* * CEF_MUST protects glimpse lock from conversion into * a lockless mode. */
@@ -137,8 +133,21 @@ int cl_glimpse_lock(const struct lu_env *env, struct cl_io *io,return result; } -static int cl_io_get (struct inode *inode,struct lu_env **envout, -struct cl_io **ioout,u16 *refcheck) +/** +* Get an IO environment for special operations such as glimpse locks and +* manually requested locks (ladvise lockahead) +* +* @inode inode the operation is being performed on +* @envout thread specific execution environment +* @ioout client io description +* @refcheck reference check +* +* Return 1 on success +* 0 not a regular file, cannot get environment +* negative errno on error +*/ +int cl_io_get (struct inode *inode,struct lu_env **envout, +struct cl_io **ioout,u16 *refcheck) {struct lu_env *env;struct cl_io *io;
fs/lustre/llite/llite_internal.h +9 -0
@@ -652,6 +652,7 @@ struct ll_file_data {* false: unknown failure, should report. */ bool fd_write_failed; +bool ll_lock_no_expand;rwlock_t fd_lock;/* protect lcc list */ struct list_head fd_lccs;/* list of ll_cl_context */ };
@@ -1163,11 +1164,19 @@ static inline int cl_glimpse_size(struct inode *inode)return __cl_glimpse_size (inode,0 ); } +/* AGL is 'asychronous glimpse lock', which is a speculative lock taken as +* part of statahead +*/ static inline int cl_agl (struct inode *inode) {return __cl_glimpse_size (inode,1 ); } +int ll_file_lock_ahead (struct file *file,struct llapi_lu_ladvise *ladvise); + +int cl_io_get (struct inode *inode,struct lu_env **envout, +struct cl_io **ioout,__u16 *refcheck); +static inline int ll_glimpse_size (struct inode *inode) {struct ll_inode_info *lli =ll_i2info (inode);
fs/lustre/llite/llite_lib.c +20 -3
@@ -185,7 +185,7 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)*/ data->ocd_grant_blkbits =PAGE_SHIFT ; -/* indicate the features supported by this client */ +/* indicate MDT features supported by this client */ data->ocd_connect_flags =OBD_CONNECT_IBITS |OBD_CONNECT_NODEVOH |OBD_CONNECT_ATTRFID | OBD_CONNECT_VERSION | OBD_CONNECT_BRW_SIZE |
@@ -374,6 +374,7 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt) goto out_md_fid; } +/* indicate OST features supported by this client */ data->ocd_connect_flags =OBD_CONNECT_GRANT |OBD_CONNECT_VERSION |OBD_CONNECT_REQPORTAL |OBD_CONNECT_BRW_SIZE | OBD_CONNECT_CANCELSET | OBD_CONNECT_FID |
@@ -386,9 +387,25 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)OBD_CONNECT_JOBSTATS |OBD_CONNECT_LVB_TYPE |OBD_CONNECT_LAYOUTLOCK |OBD_CONNECT_PINGLESS |OBD_CONNECT_LFSCK | -OBD_CONNECT_BULK_MBITS ; +OBD_CONNECT_BULK_MBITS |OBD_CONNECT_FLAGS2 ; + +/* The client currently advertises support for OBD_CONNECT_LOCKAHEAD_OLD +* so it can interoperate with an older version of lockahead which was +* released prior to landing in master. This support will be dropped +* when 2.13 development starts. At the point, we should not just drop +* the connect flag (below), we should also remove the support in the +* code. +* +* Removing it means a few things: +* 1. Remove this section here +* 2. Remove CEF_NONBLOCK in ll_file_lockahead() +* 3. Remove function exp_connect_lockahead_old +* 4. Remove LDLM_FL_LOCKAHEAD_OLD_RESERVED in lustre_dlm_flags.h +*/ +if (data->ocd_version <OBD_OCD_VERSION (2 ,12 ,50 ,0 )) + data->ocd_connect_flags |=OBD_CONNECT_LOCKAHEAD_OLD ; - data->ocd_connect_flags2 =0 ; + data->ocd_connect_flags2 =OBD_CONNECT2_LOCKAHEAD ;if (!OBD_FAIL_CHECK (OBD_FAIL_OSC_CONNECT_GRANT_PARAM )) data->ocd_connect_flags |=OBD_CONNECT_GRANT_PARAM ;
fs/lustre/llite/vvp_io.c +3 -0
@@ -524,6 +524,9 @@ static int vvp_io_rw_lock(const struct lu_env *env, struct cl_io *io,if (io->u.ci_rw.crw_nonblock) ast_flags |=CEF_NONBLOCK ; +if (io->ci_lock_no_expand) + ast_flags |=CEF_LOCK_NO_EXPAND ; + result =vvp_mmap_locks (env, vio, io);if (result ==0 ) result =vvp_io_one_lock (env, io, ast_flags, mode, start, end);
fs/lustre/lov/lov_io.c +1 -0
@@ -120,6 +120,7 @@ static int lov_io_sub_init(const struct lu_env *env, struct lov_io *lio, sub_io->ci_type = io->ci_type; sub_io->ci_no_srvlock = io->ci_no_srvlock; sub_io->ci_noatime = io->ci_noatime; + sub_io->ci_lock_no_expand = io->ci_lock_no_expand; rc =cl_io_sub_init (sub->sub_env, sub_io, io->ci_type, sub_obj);if (rc <0 )
fs/lustre/obdclass/cl_lock.c +1 -1
@@ -188,7 +188,7 @@ int cl_lock_request(const struct lu_env *env, struct cl_io *io,if (rc <0 )return rc; -if ((enq_flags &CEF_ASYNC ) && !(enq_flags &CEF_AGL )) { +if ((enq_flags &CEF_GLIMPSE ) && !(enq_flags &CEF_SPECULATIVE )) { anchor = &cl_env_info (env)->clt_anchor;cl_sync_io_init (anchor,1 ); }
fs/lustre/obdclass/lprocfs_status.c +2 -1
@@ -106,12 +106,13 @@"multi_mod_rpcs" ,"dir_stripe" ,"subtree" , -"lock_ahead" , +"lockahead" ,"bulk_mbits" ,"compact_obdo" ,"second_flags" ,/* flags2 names */ "file_secctx" , +"lockaheadv2" ,NULL };
fs/lustre/osc/osc_internal.h +2 -1
@@ -53,7 +53,8 @@ int osc_enqueue_base(struct obd_export *exp, struct ldlm_res_id *res_id, struct ost_lvb *lvb, int kms_valid, osc_enqueue_upcall_f upcall, void *cookie, struct ldlm_enqueue_info *einfo, - struct ptlrpc_request_set *rqset, int async, int agl); + struct ptlrpc_request_set *rqset, int async, + bool speculative); int osc_match_base(struct obd_export *exp, struct ldlm_res_id *res_id, enum ldlm_type type, union ldlm_policy_data *policy,
fs/lustre/osc/osc_lock.c +53 -18
160 lines not on this page · raw message
fs/lustre/osc/osc_request.c +32 -18
116 lines not on this page · raw message
fs/lustre/ptlrpc/wiretest.c +6 -2
@@ -40,6 +40,8 @@#include <obd_class.h> #include <lustre_net.h> #include <lustre_disk.h> +#include <uapi/linux/lustre/lustre_idl.h> +#include "ptlrpc_internal.h" void lustre_assert_wire_constants (void )
@@ -1113,14 +1115,16 @@ void lustre_assert_wire_constants(void) OBD_CONNECT_DIR_STRIPE);LASSERTF (OBD_CONNECT_SUBTREE ==0x800000000000000ULL ,"found 0x%.16llxULL\n" ,OBD_CONNECT_SUBTREE ); -LASSERTF (OBD_CONNECT_LOCK_AHEAD ==0x1000000000000000ULL ,"found 0x%.16llxULL\n" , -OBD_CONNECT_LOCK_AHEAD ); +LASSERTF (OBD_CONNECT_LOCKAHEAD_OLD ==0x1000000000000000ULL ,"found 0x%.16llxULL\n" , +OBD_CONNECT_LOCKAHEAD_OLD );LASSERTF (OBD_CONNECT_OBDOPACK ==0x4000000000000000ULL ,"found 0x%.16llxULL\n" ,OBD_CONNECT_OBDOPACK );LASSERTF (OBD_CONNECT_FLAGS2 ==0x8000000000000000ULL ,"found 0x%.16llxULL\n" ,OBD_CONNECT_FLAGS2 );LASSERTF (OBD_CONNECT2_FILE_SECCTX ==0x1ULL ,"found 0x%.16llxULL\n" ,OBD_CONNECT2_FILE_SECCTX ); +LASSERTF (OBD_CONNECT2_LOCKAHEAD ==0x2ULL ,"found 0x%.16llxULL\n" , +OBD_CONNECT2_LOCKAHEAD );LASSERTF (OBD_CKSUM_CRC32 ==0x00000001UL ,"found 0x%.8xUL\n" , (unsigned int )OBD_CKSUM_CRC32 ); LASSERTF(OBD_CKSUM_ADLER == 0x00000002UL, "found 0x%.8xUL\n",
include/uapi/linux/lustre/lustre_idl.h +11 -1
@@ -774,7 +774,8 @@ struct ptlrpc_body_v2 {*/ #define OBD_CONNECT_DIR_STRIPE 0x400000000000000ULL/* striped DNE dir */ #define OBD_CONNECT_SUBTREE 0x800000000000000ULL/* fileset mount */ -#define OBD_CONNECT_LOCK_AHEAD 0x1000000000000000ULL/* lock ahead */ +#define OBD_CONNECT_LOCKAHEAD_OLD 0x1000000000000000ULL/* Old Cray lockahead */ +/** bulk matchbits is sent within ptlrpc_body */ #define OBD_CONNECT_BULK_MBITS 0x2000000000000000ULL#define OBD_CONNECT_OBDOPACK 0x4000000000000000ULL/* compact OUT obdo */
@@ -783,6 +784,9 @@ struct ptlrpc_body_v2 {#define OBD_CONNECT2_FILE_SECCTX 0x1ULL/* set file security * context at create */ +#define OBD_CONNECT2_LOCKAHEAD 0x2ULL/* ladvise lockahead +* v2 +*/ /* XXX README XXX: * Please DO NOT add flag values here before first ensuring that this same
@@ -2097,6 +2101,12 @@ struct ldlm_extent {__u64 gid; }; +static inline bool ldlm_extent_equal (const struct ldlm_extent *ex1, +const struct ldlm_extent *ex2) +{ +return ex1->start == ex2->start && ex1->end == ex2->end; +} +struct ldlm_inodebits {__u64 bits; };
include/uapi/linux/lustre/lustre_user.h +36 -4
@@ -1409,11 +1409,16 @@ enum lu_ladvise_type {LU_LADVISE_INVALID =0 ,LU_LADVISE_WILLREAD =1 ,LU_LADVISE_DONTNEED =2 , +LU_LADVISE_LOCKNOEXPAND =3 , +LU_LADVISE_LOCKAHEAD =4 , + LU_LADVISE_MAX }; -#define LU_LADVISE_NAMES { \ - [LU_LADVISE_WILLREAD] = "willread", \ - [LU_LADVISE_DONTNEED] = "dontneed", \ +#define LU_LADVISE_NAMES { \ + [LU_LADVISE_WILLREAD] = "willread", \ + [LU_LADVISE_DONTNEED] = "dontneed", \ + [LU_LADVISE_LOCKNOEXPAND] = "locknoexpand", \ + [LU_LADVISE_LOCKAHEAD] = "lockahead", \ }/*
@@ -1433,10 +1438,20 @@ struct llapi_lu_ladvise {enum ladvise_flag {LF_ASYNC =0x00000001 , +LF_UNSET =0x00000002 , };#define LADVISE_MAGIC 0x1ADF1CE0 -#define LF_MASK LF_ASYNC +/* Masks of valid flags for each advice */ +#define LF_LOCKNOEXPAND_MASK LF_UNSET +/* Flags valid for all advices not explicitly specified */ +#define LF_DEFAULT_MASK LF_ASYNC +/* All flags */ +#define LF_MASK (LF_ASYNC | LF_UNSET) + +#define lla_lockahead_mode lla_value1 +#define lla_peradvice_flags lla_value2 +#define lla_lockahead_result lla_value3/* * This is the userspace argument for ladvise, corresponds to ladvise_hdr which
@@ -1455,6 +1470,23 @@ struct llapi_ladvise_hdr {#define LAH_COUNT_MAX 1024 +enum lock_mode_user { +MODE_READ_USER =1 , +MODE_WRITE_USER , +MODE_MAX_USER , +}; + +#define LOCK_MODE_NAMES { \ + [MODE_READ_USER] = "READ", \ + [MODE_WRITE_USER] = "WRITE" \ +} + +enum lockahead_results { +LLA_RESULT_SENT =0 , +LLA_RESULT_DIFFERENT , +LLA_RESULT_SAME , +}; +/** @} lustreuser */ #endif /* _LUSTRE_USER_H */
Signature · 1 line
1.8.3.1
PATCH 35/151 lustre: jobstats: move jobstats code into separate file.
In reply to James SimmonsFrom: Ben Evans <[email protected]>
We'll be enhancing jobstats in future patches. To make that easier, move the jobstats code into a separate file.
Note: the OpenSFS version of this patch added a hash table to cache content read from the environment, but we don't use environment values in this branch.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9221
Lustre-commit: 08479b74ec35 ("LU-9221 jobstats: Create a pid-based hash for jobid values")
Reviewed-on: https://review.whamcloud.com/25208
--- fs/lustre/obdclass/Makefile | 3 +- fs/lustre/obdclass/class_obd.c | 43 --------------------- fs/lustre/obdclass/jobid.c | 86 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 88 insertions(+), 44 deletions(-) create mode 100644 fs/lustre/obdclass/jobid.c
fs/lustre/obdclass/Makefile +2 -1
@@ -7,4 +7,5 @@ obdclass-y := llog.o llog_cat.o llog_obd.o llog_swab.o class_obd.o \ genops.o obd_sysfs.o lprocfs_status.o lprocfs_counters.o \ lustre_handles.o lustre_peer.o statfs_pack.o linkea.o \ obdo.o obd_config.o obd_mount.o lu_object.o lu_ref.o \ - cl_object.o cl_page.o cl_lock.o cl_io.o kernelcomm.o + cl_object.o cl_page.o cl_lock.o cl_io.o kernelcomm.o \ + jobid.o
fs/lustre/obdclass/class_obd.c +0 -43
49 lines not on this page · raw message
fs/lustre/obdclass/jobid.c new file +86 -0
86 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 36/151 lustre: ldlm: don't use jiffies as sysfs parameter
In reply to James SimmonsThe ldlm sysfs file handles lru_max_age in jiffies which is wrong since jiffies are not consistent across machine since HZ is configurable at compile time. Talking to most users they thought lru_max_age was in seconds which is incorrect. The best way to fix this is to move lru_max_age to millisecs since most systems lustre deals with sets HZ to 1000. To make it clear it is in milliseconds print out lru_max_age with "ms". Since users tend to think in seconds allow passing in seconds besides milliseconds and internally converting them to nanaseconds. Since we have to support milliseconds move to ktime_t since we can't use time64_t. Unfortunately, this makes a relatively large patch, but I could not find a way to split it up some more without breaking atomicity of the change.
WC-bug-id: https://jira.whamcloud.com/browse/LU-8541
Lustre-commit: 800ffd471186 ("LU-8541 ldlm: don't use jiffies as sysfs parameter")
Reviewed-on: https://review.whamcloud.com/28370
--- fs/lustre/include/lustre_dlm.h | 10 +++++----- fs/lustre/ldlm/ldlm_internal.h | 5 +++-- fs/lustre/ldlm/ldlm_lock.c | 7 ++++--- fs/lustre/ldlm/ldlm_lockd.c | 5 +++-- fs/lustre/ldlm/ldlm_request.c | 16 ++++++++-------- fs/lustre/ldlm/ldlm_resource.c | 23 ++++++++++++++++++----- 6 files changed, 41 insertions(+), 25 deletions(-)
fs/lustre/include/lustre_dlm.h +5 -5
@@ -59,7 +59,7 @@#define OBD_LDLM_DEVICENAME "ldlm"#define LDLM_DEFAULT_LRU_SIZE (100 * num_online_cpus()) -#define LDLM_DEFAULT_MAX_ALIVE (65 * 60 * HZ)/* 65 min */ +#define LDLM_DEFAULT_MAX_ALIVE (64 * 60)/* 65 min */ #define LDLM_DEFAULT_PARALLEL_AST_LIMIT 1024/**
@@ -410,8 +410,9 @@ struct ldlm_namespace {* controlled by available memory on this client and on server. */ unsigned int ns_max_unused; +/** Maximum allowed age (last used time) for locks in the LRU */ -unsigned int ns_max_age; +ktime_t ns_max_age;/** * Used to rate-limit ldlm_namespace_dump calls.
@@ -702,10 +703,9 @@ struct ldlm_lock {time64_t l_last_activity;/** -* Time last used by e.g. being matched by lock match. -* Jiffies. Should be converted to time if needed. +* Time, in nanoseconds, last used by e.g. being matched by lock match. */ -unsigned long l_last_used; +ktime_t l_last_used;/** Originally requested extent for the extent lock. */ struct ldlm_extent l_req_extent;
fs/lustre/ldlm/ldlm_internal.h +3 -2
@@ -147,8 +147,9 @@ void ldlm_lock_decref_internal_nolock(struct ldlm_lock *lock,struct ldlm_lock *ldlm_lock_get (struct ldlm_lock *lock);int ldlm_run_ast_work (struct ldlm_namespace *ns,struct list_head *rpc_list,enum ldlm_desc_ast_t ast_type); -int ldlm_lock_remove_from_lru_check (struct ldlm_lock *lock,time_t last_use); -#define ldlm_lock_remove_from_lru (lock) ldlm_lock_remove_from_lru_check(lock, 0) +int ldlm_lock_remove_from_lru_check (struct ldlm_lock *lock,ktime_t last_use); +#define ldlm_lock_remove_from_lru (lock) \ + ldlm_lock_remove_from_lru_check(lock, ktime_set(0, 0))int ldlm_lock_remove_from_lru_nolock (struct ldlm_lock *lock);void ldlm_lock_destroy_nolock (struct ldlm_lock *lock);
fs/lustre/ldlm/ldlm_lock.c +4 -3
@@ -224,13 +224,14 @@ int ldlm_lock_remove_from_lru_nolock(struct ldlm_lock *lock)* * 1 the lock was in LRU list and removed. */ -int ldlm_lock_remove_from_lru_check (struct ldlm_lock *lock,time_t last_use) +int ldlm_lock_remove_from_lru_check (struct ldlm_lock *lock,ktime_t last_use) {struct ldlm_namespace *ns =ldlm_lock_to_ns (lock);int rc =0 ;spin_lock (&ns->ns_lock); -if (last_use ==0 || last_use == lock->l_last_used) +if (!ktime_compare (last_use,ktime_set (0 ,0 )) || + !ktime_compare (last_use, lock->l_last_used)) rc =ldlm_lock_remove_from_lru_nolock (lock);spin_unlock (&ns->ns_lock);
@@ -244,7 +245,7 @@ static void ldlm_lock_add_to_lru_nolock(struct ldlm_lock *lock) {struct ldlm_namespace *ns =ldlm_lock_to_ns (lock); - lock->l_last_used = jiffies; + lock->l_last_used =ktime_get ();LASSERT (list_empty (&lock->l_lru));LASSERT (lock->l_resource->lr_type !=LDLM_FLOCK );list_add_tail (&lock->l_lru, &ns->ns_unused_list);
fs/lustre/ldlm/ldlm_lockd.c +3 -2
@@ -303,8 +303,9 @@ static void ldlm_handle_gl_callback(struct ptlrpc_request *req,lock_res_and_lock (lock);if (lock->l_granted_mode ==LCK_PW && !lock->l_readers && !lock->l_writers && -time_after (jiffies, - lock->l_last_used +10 *HZ )) { +ktime_after (ktime_get (), +ktime_add (lock->l_last_used, +ktime_set (10 ,0 )))) {unlock_res_and_lock (lock);if (ldlm_bl_to_thread_lock (ns,NULL , lock))ldlm_handle_bl_callback (ns,NULL , lock);
fs/lustre/ldlm/ldlm_request.c +8 -8
@@ -1166,10 +1166,10 @@ static enum ldlm_policy_res ldlm_cancel_lrur_policy(struct ldlm_namespace *ns, int unused, int added, int count) { - unsigned long cur = jiffies; + ktime_t cur = ktime_get();struct ldlm_pool *pl = &ns->ns_pool;u64 slv, lvf, lv; -unsigned long la; +s64 la;/* Stop LRU processing when we reach past @count or have checked all * locks in LRU.
@@ -1181,12 +1181,13 @@ static enum ldlm_policy_res ldlm_cancel_lrur_policy(struct ldlm_namespace *ns,* Despite of the LV, It doesn't make sense to keep the lock which * is unused for ns_max_age time. */ -if (time_after (jiffies, lock->l_last_used + ns->ns_max_age)) +if (ktime_after (ktime_get (), +ktime_add (lock->l_last_used, ns->ns_max_age)))return LDLM_POLICY_CANCEL_LOCK ; slv =ldlm_pool_get_slv (pl); lvf =ldlm_pool_get_lvf (pl); - la = (cur - lock->l_last_used) /HZ ; + la =ktime_to_ns (ktime_sub (cur, lock->l_last_used)) /NSEC_PER_SEC ; lv = lvf * la * unused;/* Inform pool about current CLV to see it via debugfs. */
@@ -1235,7 +1236,8 @@ static enum ldlm_policy_res ldlm_cancel_aged_policy(struct ldlm_namespace *ns, int count) {if ((added >= count) && -time_before (jiffies, lock->l_last_used + ns->ns_max_age)) +ktime_before (ktime_get (), +ktime_add (lock->l_last_used, ns->ns_max_age)))return LDLM_POLICY_KEEP_LOCK ;return LDLM_POLICY_CANCEL_LOCK ;
@@ -1382,7 +1384,7 @@ static int ldlm_prepare_lru_list(struct ldlm_namespace *ns,while (!list_empty (&ns->ns_unused_list)) {enum ldlm_policy_res result; -time_t last_use =0 ; +ktime_t last_use =ktime_set (0 ,0 );/* all unused locks */ if (remained-- <=0 )
@@ -1402,8 +1404,6 @@ static int ldlm_prepare_lru_list(struct ldlm_namespace *ns,continue ; last_use = lock->l_last_used; -if (last_use == jiffies) -continue ;/* Somebody is already doing CANCEL. No need for this * lock in LRU, do not traverse it again.
fs/lustre/ldlm/ldlm_resource.c +18 -5
@@ -263,7 +263,7 @@ static ssize_t lru_max_age_show(struct kobject *kobj, struct attribute *attr, struct ldlm_namespace *ns = container_of(kobj, struct ldlm_namespace, ns_kobj); -return sprintf (buf,"%u\n" , ns->ns_max_age); +return sprintf (buf,"%lld\n" ,ktime_to_ms (ns->ns_max_age)); } static ssize_t lru_max_age_store(struct kobject *kobj, struct attribute *attr,
@@ -271,14 +271,27 @@ static ssize_t lru_max_age_store(struct kobject *kobj, struct attribute *attr, { struct ldlm_namespace *ns = container_of(kobj, struct ldlm_namespace, ns_kobj); -unsigned long tmp; +int scale =NSEC_PER_MSEC ; +unsigned long long tmp; +char *buf;int err; - err =kstrtoul (buffer,10 , &tmp); +/* Did the user ask in seconds or milliseconds. Default is in ms */ + buf =strstr (buffer,"ms" ); +if (!buf) { + buf =strchr (buffer,'s' ); +if (buf) + scale =NSEC_PER_SEC ; + } + +if (buf) + *buf ='\0' ; + + err =kstrtoull (buffer,10 , &tmp);if (err !=0 )return -EINVAL ; - ns->ns_max_age = tmp; + ns->ns_max_age =ktime_set (0 , tmp * scale);return count; }
@@ -639,7 +652,7 @@ struct ldlm_namespace *ldlm_namespace_new(struct obd_device *obd, char *name, ns->ns_max_parallel_ast =LDLM_DEFAULT_PARALLEL_AST_LIMIT ; ns->ns_nr_unused =0 ; ns->ns_max_unused =LDLM_DEFAULT_LRU_SIZE ; - ns->ns_max_age =LDLM_DEFAULT_MAX_ALIVE ; + ns->ns_max_age =ktime_set (LDLM_DEFAULT_MAX_ALIVE ,0 ); ns->ns_orig_connect_flags =0 ; ns->ns_connect_flags =0 ; ns->ns_stopping =0 ;
Signature · 1 line
1.8.3.1
PATCH 37/151 lnet: Handle ping buffer with only loopback NID
In reply to James SimmonsFrom: Olaf Weber <[email protected]>
During startup lnet_peer_data_resent() can see a ping buffer for the local node which contains only the loopback NID. This shows up as pi_nnis == 1, and there is nothing to be done (or that needs to be done) in that case.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9933
Lustre-commit: 002e25b92779 ("LU-9933 lnet: Handle ping buffer with only loopback NID")
Reviewed-on: https://review.whamcloud.com/28811
--- net/lnet/lnet/peer.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
net/lnet/lnet/peer.c +5 -3
@@ -2589,10 +2589,12 @@ static int lnet_peer_data_present(struct lnet_peer *lp)* * The peer for the loopback interface is a special case: this * is the peer for the local node, and we want to set its -* primary NID to the correct value here. +* primary NID to the correct value here. Moreover, this peer +* can show up with only the loopback NID in the ping buffer. */ -if (pbuf->pb_info.pi_nnis >1 ) - nid = pbuf->pb_info.pi_ni[1 ].ns_nid; +if (pbuf->pb_info.pi_nnis <=1 ) + goto out; + nid = pbuf->pb_info.pi_ni[1 ].ns_nid;if (LNET_NETTYP (LNET_NIDNET (lp->lp_primary_nid)) ==LOLND ) { rc =lnet_peer_set_primary_nid (lp, nid, flags);if (!rc)
Signature · 1 line
1.8.3.1
PATCH 38/151 lustre: llite: enable readahead for small read_ahead_per_file
In reply to James SimmonsFrom: Erich Focht <[email protected]>
Fixes for a regression introduced by http://review.whamcloud.com/19368
for the case that max_read_ahead_per_file_mb is smaller than max_pages_per_rpc. With 16MB RPCs this happens pretty easily. In that case the readahead window stayed zero and the backend saw only requests of the size of the user IOs.
This patch restores the previous behavior for this corner case while keeping the fix for large RPCs introduced by the alignment. When max_read_ahead_per_file_mb is smaller than max_pages_per_rpc the RPC size will not be optimal, but will be at least 1MB and the readahead window will be as large as expected instead of zero.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9214
Lustre-commit: 32e64eeefa7d ("LU-9214 llite: enable readahead for small read_ahead_per_file")
Reviewed-on: https://review.whamcloud.com/25996
--- fs/lustre/llite/rw.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
fs/lustre/llite/rw.c +4 -1
@@ -719,7 +719,10 @@ static void ras_increase_window(struct inode *inode, wlen =min (ras->ras_window_len + ras->ras_rpc_size, ra->ra_max_pages_per_file); - ras->ras_window_len =ras_align (ras, wlen,NULL ); +if (wlen < ras->ras_rpc_size) + ras->ras_window_len = wlen; +else + ras->ras_window_len =ras_align (ras, wlen,NULL ); } }
Signature · 1 line
1.8.3.1
PATCH 39/151 lnet: don't discover loopback interface
In reply to James SimmonsFrom: Amir Shehata <[email protected]>
Whenever we send messages destined to the loopback interface it should always go over the loopback interface. To achieve that there is no real need to initiate discovery on the loopback. This will result in a non-mr peer created for the loopback, Which makes sense because if we are to send messages to ourselves we do not want to use the different interfaces rather just keep sending over the lolnd. In effect this is a special case where we want to behave as a non-mr node.
When sending a message destined for the loopback interface there is no need to go through the selection process, it is sufficient to shortcut all the MR logic and send directly over the lolnd.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9992
Lustre-commit: de823879c43d ("LU-9992 lnet: don't discover loopback interface")
Reviewed-on: https://review.whamcloud.com/29007
--- net/lnet/lnet/lib-move.c | 11 +++++++++++ net/lnet/lnet/peer.c | 9 +++++++++ 2 files changed, 20 insertions(+)
net/lnet/lnet/lib-move.c +11 -0
@@ -1391,6 +1391,17 @@ void lnet_usr_translate_stats(struct lnet_ioctl_element_msg_stats *msg_stats,lnet_net_unlock (cpt);return PTR_ERR (lpni); } + +/* If we're being asked to send to the loopback interface, there +* is no need to go through any selection. We can just shortcut +* the entire process and send over lolnd +*/ +if (LNET_NETTYP (LNET_NIDNET (dst_nid)) ==LOLND ) { +lnet_peer_ni_decref_locked (lpni); + best_ni = the_lnet.ln_loni; + goto send; + } +/* * Now that we have a peer_ni, check if we want to discover * the peer. Traffic to the LNET_RESERVED_PORTAL should not
net/lnet/lnet/peer.c +9 -0
@@ -211,6 +211,15 @@init_waitqueue_head (&lp->lp_dc_waitq);spin_lock_init (&lp->lp_lock); lp->lp_primary_nid = nid; + +/* Turn off discovery for loopback peer. If you're creating a peer +* for the loopback interface then that was initiated when we +* attempted to send a message over the loopback. There is no need +* to ever use a different interface when sending messages to +* myself. +*/ +if (LNET_NETTYP (LNET_NIDNET (nid)) ==LOLND ) + lp->lp_state =LNET_PEER_NO_DISCOVERY ; lp->lp_cpt =lnet_nid_cpt_hash (nid,LNET_CPT_NUMBER );CDEBUG (D_NET ,"%p nid %s\n" , lp,libcfs_nid2str (lp->lp_primary_nid));
Signature · 1 line
1.8.3.1
PATCH 40/151 lnet: reduce logging severity
In reply to James SimmonsFrom: Amir Shehata <[email protected]>
On shutdown a push event can be triggered for a non-existent peer. Reducing the severity of the log.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9904
Lustre-commit: 7fc8037d61b2 ("LU-9904 lnet: reduce logging severity")
Reviewed-on: https://review.whamcloud.com/29026
--- net/lnet/lnet/peer.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
net/lnet/lnet/peer.c +2 -1
@@ -1808,7 +1808,8 @@ void lnet_peer_push_event(struct lnet_event *ev)/* lnet_find_peer() adds a refcount */ lp =lnet_find_peer (ev->source.nid);if (!lp) { -CERROR ("Push Put from unknown %s (source %s)\n" , +CDEBUG (D_NET , +"Push Put from unknown %s (source %s). Ignoring...\n" ,libcfs_nid2str (ev->initiator.nid),libcfs_nid2str (ev->source.nid));return ;
Signature · 1 line
1.8.3.1
PATCH 41/151 lustre: ptlrpc: migrate pinger to 64 bit time
In reply to James SimmonsChange imp_next_ping, cl_*grant_* timer fields, and ti_timeout to time64_t. With these changes the pinger will be 64 bit time compliant. This prevents any possible confusion with jiffies.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9019
Lustre-commit: 44bdc137c901 ("LU-9019 ptlrpc: migrate pinger to 64 bit time")
Reviewed-on: https://review.whamcloud.com/28035
--- fs/lustre/include/lustre_import.h | 2 +- fs/lustre/include/lustre_net.h | 2 +- fs/lustre/include/obd.h | 6 +++--- fs/lustre/osc/lproc_osc.c | 2 +- fs/lustre/osc/osc_request.c | 11 +++++----- fs/lustre/ptlrpc/client.c | 7 +++--- fs/lustre/ptlrpc/import.c | 36 +++++++++++++++++++------------ fs/lustre/ptlrpc/pinger.c | 44 ++++++++++++++++++-------------------- fs/lustre/ptlrpc/ptlrpc_internal.h | 2 +- fs/lustre/ptlrpc/ptlrpcd.c | 2 +- 10 files changed, 60 insertions(+), 54 deletions(-)
fs/lustre/include/lustre_import.h +1 -1
@@ -250,7 +250,7 @@ struct obd_import {*/ struct lustre_handle imp_remote_handle;/** When to perform next ping. time in jiffies. */ -unsigned long imp_next_ping; +time64_t imp_next_ping;/** When we last successfully connected. time in 64bit jiffies */ u64 imp_last_success_conn;
fs/lustre/include/lustre_net.h +1 -1
@@ -2287,7 +2287,7 @@ enum timeout_event {typedef int (*timeout_cb_t )(struct timeout_item *,void *);int ptlrpc_pinger_add_import (struct obd_import *imp);int ptlrpc_pinger_del_import (struct obd_import *imp); -int ptlrpc_add_timeout_client (int time,enum timeout_event event, +int ptlrpc_add_timeout_client (time64_t time,enum timeout_event event,timeout_cb_t cb,void *data,struct list_head *obd_list); int ptlrpc_del_timeout_client(struct list_head *obd_list,
fs/lustre/include/obd.h +3 -3
@@ -117,7 +117,7 @@ struct brw_page {struct timeout_item {enum timeout_event ti_event; -unsigned long ti_timeout; +time64_t ti_timeout;timeout_cb_t ti_cb;void *ti_cb_data;struct list_head ti_obd_list;
@@ -203,9 +203,9 @@ struct client_obd {*/ long cl_reserved_grant;wait_queue_head_t cl_cache_waiters;/* waiting for cache/grant */ -unsigned long cl_next_shrink_grant;/* jiffies */ +time64_t cl_next_shrink_grant;/* seconds */ struct list_head cl_grant_shrink_list;/* Timeout event list */ -int cl_grant_shrink_interval;/* seconds */ +time64_t cl_grant_shrink_interval;/* seconds */ /* A chunk is an optimal size used by osc_extent to determine * the extent size. A chunk is max(PAGE_SIZE, OST block size)
fs/lustre/osc/lproc_osc.c +1 -1
@@ -326,7 +326,7 @@ static ssize_t grant_shrink_interval_show(struct kobject *kobj, struct obd_device *obd = container_of(kobj, struct obd_device, obd_kset.kobj); -return sprintf (buf,"%d\n" , obd->u.cli.cl_grant_shrink_interval); +return sprintf (buf,"%lld\n" , obd->u.cli.cl_grant_shrink_interval); } static ssize_t grant_shrink_interval_store(struct kobject *kobj,
fs/lustre/osc/osc_request.c +5 -6
@@ -716,9 +716,9 @@ static void osc_announce_cached(struct client_obd *cli, struct obdo *oa,void osc_update_next_shrink (struct client_obd *cli) { - cli->cl_next_shrink_grant = - jiffies + cli->cl_grant_shrink_interval *HZ ; -CDEBUG (D_CACHE ,"next time %ld to shrink grant\n" , + cli->cl_next_shrink_grant =ktime_get_seconds () + + cli->cl_grant_shrink_interval; +CDEBUG (D_CACHE ,"next time %lld to shrink grant\n" , cli->cl_next_shrink_grant); }
@@ -841,14 +841,13 @@ int osc_shrink_grant_to_target(struct client_obd *cli, u64 target_bytes)static int osc_should_shrink_grant (struct client_obd *client) { -unsigned long time = jiffies; -unsigned long next_shrink = client->cl_next_shrink_grant; +time64_t next_shrink = client->cl_next_shrink_grant;if ((client->cl_import->imp_connect_data.ocd_connect_flags &OBD_CONNECT_GRANT_SHRINK ) ==0 )return 0 ; -if (time_after_eq (time, next_shrink -5 )) { +if (ktime_get_seconds () >= next_shrink -5 ) {/* Get the current RPC size directly, instead of going via: * cli_brw_size(obd->u.cli.cl_import->imp_obd->obd_self_export) * Keep comment here so that it can be found by searching.
fs/lustre/ptlrpc/client.c +4 -3
@@ -2206,7 +2206,7 @@ static void ptlrpc_interrupted_set(struct ptlrpc_request_set *set)/** * Get the smallest timeout in the set; this does NOT set a timeout. */ -int ptlrpc_set_next_timeout (struct ptlrpc_request_set *set) +time64_t ptlrpc_set_next_timeout (struct ptlrpc_request_set *set) {time64_t now =ktime_get_real_seconds ();int timeout =0 ;
@@ -2252,7 +2252,8 @@ int ptlrpc_set_next_timeout(struct ptlrpc_request_set *set)int ptlrpc_set_wait (struct ptlrpc_request_set *set) {struct ptlrpc_request *req; -int rc, timeout; +time64_t timeout; +int rc;if (set->set_producer) (void )ptlrpc_set_producer (set);
@@ -2272,7 +2273,7 @@ int ptlrpc_set_wait(struct ptlrpc_request_set *set)* wait until all complete, interrupted, or an in-flight * req times out */ -CDEBUG (D_RPCTRACE ,"set %p going to sleep for %d seconds\n" , +CDEBUG (D_RPCTRACE ,"set %p going to sleep for %lld seconds\n" , set, timeout);if (timeout ==0 && !signal_pending (current)) {
fs/lustre/ptlrpc/import.c +22 -14
@@ -218,10 +218,10 @@ void ptlrpc_deactivate_import(struct obd_import *imp) }EXPORT_SYMBOL (ptlrpc_deactivate_import); -static unsigned int +static time64_t ptlrpc_inflight_deadline (struct ptlrpc_request *req,time64_t now) { -long dl; +time64_t dl; if (!(((req->rq_phase == RQ_PHASE_RPC) && !req->rq_waiting) || (req->rq_phase == RQ_PHASE_BULK) ||
@@ -246,7 +246,7 @@ static unsigned int ptlrpc_inflight_timeout(struct obd_import *imp) {time64_t now =ktime_get_real_seconds ();struct ptlrpc_request *req, *n; -unsigned int timeout =0 ; +time64_t timeout =0 ;spin_lock (&imp->imp_lock);list_for_each_entry_safe (req, n, &imp->imp_sending_list, rq_list)
@@ -265,7 +265,7 @@ static unsigned int ptlrpc_inflight_timeout(struct obd_import *imp)void ptlrpc_invalidate_import (struct obd_import *imp) {struct ptlrpc_request *req, *n; -unsigned int timeout; +time64_t timeout;int rc;atomic_inc (&imp->imp_inval_count);
@@ -282,6 +282,7 @@ void ptlrpc_invalidate_import(struct obd_import *imp)* no guarantee that some rdma transfer is not in progress right now. */ do { + long timeout_jiffies; /* Calculate max timeout for waiting on rpcs to error * out. Use obd_timeout if calculated value is smaller * than it.
@@ -298,16 +299,17 @@ void ptlrpc_invalidate_import(struct obd_import *imp) }CDEBUG (D_RPCTRACE , -"Sleeping %d sec for inflight to error out\n" , +"Sleeping %llds for inflight to error out\n" , timeout);/* Wait for all requests to error out and call completion * callbacks. Cap it at obd_timeout -- these should all * have been locally cancelled by ptlrpc_abort_inflight. */ + timeout_jiffies =max_t (long, timeout *HZ ,1 ); rc =wait_event_idle_timeout (imp->imp_recovery_waitq,atomic_read (&imp->imp_inflight) ==0 , - obd_timeout *HZ ); + timeout_jiffies);if (rc ==0 ) {const char *cli_tgt =obd2cli_tgt (imp->imp_obd);
@@ -421,6 +423,7 @@ void ptlrpc_fail_import(struct obd_import *imp, u32 conn_cnt)int ptlrpc_reconnect_import (struct obd_import *imp) { +long timeout_jiffies = obd_timeout *HZ ;int rc;ptlrpc_pinger_force (imp);
@@ -430,7 +433,7 @@ int ptlrpc_reconnect_import(struct obd_import *imp) rc =wait_event_idle_timeout (imp->imp_recovery_waitq, !ptlrpc_import_in_recovery (imp), - obd_timeout *HZ ); + timeout_jiffies);CDEBUG (D_HA ,"%s: recovery finished s:%s\n" ,obd2cli_tgt (imp->imp_obd),ptlrpc_import_state_name (imp->imp_state));return rc ==0 ? -ETIMEDOUT :0 ;
@@ -1506,22 +1509,27 @@ int ptlrpc_disconnect_import(struct obd_import *imp, int noclose) }if (ptlrpc_import_in_recovery (imp)) { -unsigned long timeout; +long timeout_jiffies; +time64_t timeout;if (AT_OFF ) {if (imp->imp_server_timeout) - timeout = obd_timeout *HZ /2 ; + timeout = obd_timeout >>1 ;else - timeout = obd_timeout *HZ ; + timeout = obd_timeout; }else { -int idx =import_at_get_index (imp, - imp->imp_client->cli_request_portal); - timeout =at_get (&imp->imp_at.iat_service_estimate[idx]) *HZ ; +u32 req_portal; +int idx; + + req_portal = imp->imp_client->cli_request_portal; + idx =import_at_get_index (imp, req_portal); + timeout =at_get (&imp->imp_at.iat_service_estimate[idx]); } + timeout_jiffies = timeout *HZ ;if (wait_event_idle_timeout (imp->imp_recovery_waitq, !ptlrpc_import_in_recovery (imp), -max (timeout,1UL )) ==0 ) +max_t (long, timeout_jiffies,1 )) ==0 )l_wait_event_abortable ( imp->imp_recovery_waitq, !ptlrpc_import_in_recovery (imp));
fs/lustre/ptlrpc/pinger.c +21 -23
127 lines not on this page · raw message
fs/lustre/ptlrpc/ptlrpc_internal.h +1 -1
@@ -69,7 +69,7 @@ struct ptlrpc_bulk_desc *ptlrpc_new_bulk(unsigned int nfrags,void ptlrpc_set_add_new_req (struct ptlrpcd_ctl *pc,struct ptlrpc_request *req);void ptlrpc_expired_set (struct ptlrpc_request_set *set); -int ptlrpc_set_next_timeout (struct ptlrpc_request_set *set); +time64_t ptlrpc_set_next_timeout (struct ptlrpc_request_set *set);void ptlrpc_resend_req (struct ptlrpc_request *request);void ptlrpc_set_bulk_mbits (struct ptlrpc_request *req);void ptlrpc_assign_next_xid_nolock (struct ptlrpc_request *req);
fs/lustre/ptlrpc/ptlrpcd.c +1 -1
@@ -434,7 +434,7 @@ static int ptlrpcd(void *arg)* new_req_list and ptlrpcd_check() moves them into the set. */ do { - int timeout; + time64_t timeout; timeout = ptlrpc_set_next_timeout(set);
Signature · 1 line
1.8.3.1
PATCH 42/151 lustre: mdc: add cl_device to the MDC
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Introduce cl_device for MDC so it is able to prepare and send IO requests to the MDT.
This patch adds basic interfaces for device and object methods. The common code is exported from OSC for MDC.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 68ff85b8858e ("LU-3285 mdc: add cl_device to the MDC")
Reviewed-on: https://review.whamcloud.com/28009
--- fs/lustre/include/lustre_osc.h | 23 +++++ fs/lustre/mdc/Makefile | 1 + fs/lustre/mdc/mdc_dev.c | 192 +++++++++++++++++++++++++++++++++++++++++ fs/lustre/mdc/mdc_internal.h | 6 ++ fs/lustre/mdc/mdc_request.c | 6 +- fs/lustre/osc/osc_dev.c | 22 +++-- fs/lustre/osc/osc_object.c | 29 ++++--- 7 files changed, 257 insertions(+), 22 deletions(-) create mode 100644 fs/lustre/mdc/mdc_dev.c
fs/lustre/include/lustre_osc.h +23 -0
@@ -489,6 +489,29 @@ void osc_io_unplug(const struct lu_env *env, struct client_obd *cli,int osc_lock_is_lockless (const struct osc_lock *olck); +/* osc_dev.c */ +int osc_device_init (const struct lu_env *env,struct lu_device *d, +const char *name,struct lu_device *next); +struct lu_device *osc_device_fini (const struct lu_env *env, +struct lu_device *d); +struct lu_device *osc_device_free (const struct lu_env *env, +struct lu_device *d); + +/* osc_object.c */ +int osc_object_init (const struct lu_env *env,struct lu_object *obj, +const struct lu_object_conf *conf); +void osc_object_free (const struct lu_env *env,struct lu_object *obj); +int osc_lvb_print (const struct lu_env *env,void *cookie, +lu_printer_t p,const struct ost_lvb *lvb); +int osc_object_print (const struct lu_env *env,void *cookie, +lu_printer_t p,const struct lu_object *obj); +int osc_attr_get (const struct lu_env *env,struct cl_object *obj, +struct cl_attr *attr); +int osc_attr_update (const struct lu_env *env,struct cl_object *obj, +const struct cl_attr *attr,unsigned int valid); +int osc_object_glimpse (const struct lu_env *env,const struct cl_object *obj, +struct ost_lvb *lvb); +/**************************************************************************** * * Accessors and type conversions.
fs/lustre/mdc/Makefile +1 -0
@@ -2,3 +2,4 @@ ccflags-y += -I$(srctree)/$(src)/../include obj-$(CONFIG_LUSTRE_FS) += mdc.o mdc-y := mdc_changelog.o mdc_request.o mdc_reint.o mdc_lib.o mdc_locks.o lproc_mdc.o +mdc-y += mdc_dev.o
fs/lustre/mdc/mdc_dev.c new file +192 -0
192 lines not on this page · raw message
fs/lustre/mdc/mdc_internal.h +6 -0
@@ -90,6 +90,9 @@ int mdc_resource_get_unused(struct obd_export *exp, const struct lu_fid *fid,/* mdc/mdc_request.c */ int mdc_fid_alloc (const struct lu_env *env,struct obd_export *exp,struct lu_fid *fid,struct md_op_data *op_data); +int mdc_setup (struct obd_device *obd,struct lustre_cfg *cfg); +int mdc_process_config (struct obd_device *obd,u32 len,void *buf); +struct obd_client_handle ; int mdc_set_open_replay_data(struct obd_export *exp,
@@ -149,4 +152,7 @@ static inline unsigned long hash_x_index(u64 hash, int hash64) return ~0UL - (hash + !hash); } +/* mdc_dev.c */ +extern struct lu_device_type mdc_device_type; +#endif
fs/lustre/mdc/mdc_request.c +3 -3
@@ -2473,7 +2473,7 @@ static void mdc_llog_finish(struct obd_device *obd)llog_cleanup (NULL , ctxt); } -static int mdc_setup (struct obd_device *obd,struct lustre_cfg *cfg) +int mdc_setup (struct obd_device *obd,struct lustre_cfg *cfg) {int rc;
@@ -2566,7 +2566,7 @@ static int mdc_cleanup(struct obd_device *obd)return client_obd_cleanup (obd); } -static int mdc_process_config (struct obd_device *obd,u32 len,void *buf) +int mdc_process_config (struct obd_device *obd,u32 len,void *buf) {struct lustre_cfg *lcfg = buf; size_t count = class_modify_config(lcfg, PARAM_MDC,
@@ -2635,7 +2635,7 @@ static int __init mdc_init(void)return rc;return class_register_type (&mdc_obd_ops, &mdc_md_ops, -LUSTRE_MDC_NAME ,NULL ); +LUSTRE_MDC_NAME , &mdc_device_type); }static void /*__exit*/ mdc_exit (void )
fs/lustre/osc/osc_dev.c +14 -8
@@ -48,11 +48,16 @@*/ struct kmem_cache *osc_lock_kmem; +EXPORT_SYMBOL (osc_lock_kmem);struct kmem_cache *osc_object_kmem; +EXPORT_SYMBOL (osc_object_kmem); +struct kmem_cache *osc_thread_kmem;struct kmem_cache *osc_session_kmem;struct kmem_cache *osc_extent_kmem; +EXPORT_SYMBOL (osc_extent_kmem);struct kmem_cache *osc_quota_kmem; +EXPORT_SYMBOL (osc_quota_kmem);struct lu_kmem_descr osc_caches[] = { {
@@ -162,20 +167,20 @@ static int osc_cl_process_config(const struct lu_env *env, .ldo_recovery_complete = NULL }; -static int osc_device_init (const struct lu_env *env,struct lu_device *d, -const char *name,struct lu_device *next) +int osc_device_init (const struct lu_env *env,struct lu_device *d, +const char *name,struct lu_device *next) {return 0 ; } - -static struct lu_device *osc_device_fini (const struct lu_env *env, -struct lu_device *d) +EXPORT_SYMBOL (osc_device_init); +struct lu_device *osc_device_fini (const struct lu_env *env, +struct lu_device *d) {return NULL ; } - -static struct lu_device *osc_device_free (const struct lu_env *env, -struct lu_device *d) +EXPORT_SYMBOL (osc_device_fini); +struct lu_device *osc_device_free (const struct lu_env *env, +struct lu_device *d) {struct osc_device *od =lu2osc_dev (d);
@@ -183,6 +188,7 @@ static struct lu_device *osc_device_free(const struct lu_env *env,kfree (od);return NULL ; } +EXPORT_SYMBOL (osc_device_free); static struct lu_device *osc_device_alloc(const struct lu_env *env, struct lu_device_type *t,
fs/lustre/osc/osc_object.c +18 -11
@@ -51,8 +51,8 @@* */ -static int osc_object_init (const struct lu_env *env,struct lu_object *obj, -const struct lu_object_conf *conf) +int osc_object_init (const struct lu_env *env,struct lu_object *obj, +const struct lu_object_conf *conf) {struct osc_object *osc =lu2osc (obj);const struct cl_object_conf *cconf =lu2cl_conf (conf);
@@ -82,8 +82,9 @@ static int osc_object_init(const struct lu_env *env, struct lu_object *obj,return 0 ; } +EXPORT_SYMBOL (osc_object_init); -static void osc_object_free (const struct lu_env *env,struct lu_object *obj) +void osc_object_free (const struct lu_env *env,struct lu_object *obj) {struct osc_object *osc =lu2osc (obj);
@@ -105,6 +106,7 @@ static void osc_object_free(const struct lu_env *env, struct lu_object *obj)lu_object_fini (obj);kmem_cache_free (osc_object_kmem, osc); } +EXPORT_SYMBOL (osc_object_free);int osc_lvb_print (const struct lu_env *env,void *cookie,lu_printer_t p,const struct ost_lvb *lvb)
@@ -114,9 +116,10 @@ int osc_lvb_print(const struct lu_env *env, void *cookie, lvb->lvb_size, lvb->lvb_mtime, lvb->lvb_atime, lvb->lvb_ctime, lvb->lvb_blocks); } +EXPORT_SYMBOL (osc_lvb_print); -static int osc_object_print (const struct lu_env *env,void *cookie, -lu_printer_t p,const struct lu_object *obj) +int osc_object_print (const struct lu_env *env,void *cookie, +lu_printer_t p,const struct lu_object *obj) {struct osc_object *osc =lu2osc (obj);struct lov_oinfo *oinfo = osc->oo_oinfo;
@@ -130,9 +133,10 @@ static int osc_object_print(const struct lu_env *env, void *cookie,osc_lvb_print (env, cookie, p, &oinfo->loi_lvb);return 0 ; } +EXPORT_SYMBOL (osc_object_print); -static int osc_attr_get (const struct lu_env *env,struct cl_object *obj, -struct cl_attr *attr) +int osc_attr_get (const struct lu_env *env,struct cl_object *obj, +struct cl_attr *attr) {struct lov_oinfo *oinfo =cl2osc (obj)->oo_oinfo;
@@ -140,9 +144,10 @@ static int osc_attr_get(const struct lu_env *env, struct cl_object *obj, attr->cat_kms = oinfo->loi_kms_valid ? oinfo->loi_kms :0 ;return 0 ; } +EXPORT_SYMBOL (osc_attr_get); -static int osc_attr_update (const struct lu_env *env,struct cl_object *obj, -const struct cl_attr *attr,unsigned int valid) +int osc_attr_update (const struct lu_env *env,struct cl_object *obj, +const struct cl_attr *attr,unsigned int valid) {struct lov_oinfo *oinfo =cl2osc (obj)->oo_oinfo;struct ost_lvb *lvb = &oinfo->loi_lvb;
@@ -164,9 +169,10 @@ static int osc_attr_update(const struct lu_env *env, struct cl_object *obj, }return 0 ; } +EXPORT_SYMBOL (osc_attr_update); -static int osc_object_glimpse (const struct lu_env *env, -const struct cl_object *obj,struct ost_lvb *lvb) +int osc_object_glimpse (const struct lu_env *env, +const struct cl_object *obj,struct ost_lvb *lvb) {struct lov_oinfo *oinfo =cl2osc (obj)->oo_oinfo;
@@ -174,6 +180,7 @@ static int osc_object_glimpse(const struct lu_env *env, lvb->lvb_blocks = oinfo->loi_lvb.lvb_blocks;return 0 ; } +EXPORT_SYMBOL (osc_object_glimpse);static int osc_object_ast_clear (struct ldlm_lock *lock,void *data) {
Signature · 1 line
1.8.3.1
PATCH 43/151 lustre: lov: add MDT target to the LOV device
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
MDC becomes LOV target like OSC for Data-on-MDT needs. Patch does the following:
- new composite layout entry type is added - LLT_DOM to describe Data-on-MDT striping.
- LOV process config log and checks for MDC targets organizing them separately from OSCs
- LOV operations are changed where needed to understand new layout entry type
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 8b352709a66f ("LU-3285 lov: add MDT target to the LOV device")
Reviewed-on: https://review.whamcloud.com/28010
--- fs/lustre/include/obd.h | 8 + fs/lustre/lmv/lmv_obd.c | 2 +- fs/lustre/lov/lov_cl_internal.h | 76 +++- fs/lustre/lov/lov_dev.c | 276 +++++++++++-- fs/lustre/lov/lov_ea.c | 20 +- fs/lustre/lov/lov_internal.h | 7 + fs/lustre/lov/lov_io.c | 6 +- fs/lustre/lov/lov_obd.c | 39 +- fs/lustre/lov/lov_object.c | 696 +++++++++++++++++++++----------- fs/lustre/lov/lov_offset.c | 3 + fs/lustre/mdc/mdc_request.c | 7 +- fs/lustre/obdclass/obd_config.c | 36 +- fs/lustre/ptlrpc/wiretest.c | 4 +- include/uapi/linux/lustre/lustre_user.h | 2 +- 14 files changed, 883 insertions(+), 299 deletions(-)
fs/lustre/include/obd.h +8 -0
@@ -381,6 +381,11 @@ struct lov_tgt_desc { ltd_reap:1 ;/* should this target be deleted */ }; +struct lov_md_tgt_desc { +struct obd_device *lmtd_mdc; +u32 lmtd_index; +}; +struct lov_obd {struct lov_desc desc;struct lov_tgt_desc **lov_tgts;/* sparse array */
@@ -403,10 +408,13 @@ struct lov_obd {struct rw_semaphore lov_notify_lock;struct kobject *lov_tgts_kobj; +/* Data-on-MDT: MDC array */ +struct lov_md_tgt_desc *lov_mdc_tgts; };struct lmv_tgt_desc {struct obd_uuid ltd_uuid; +struct obd_device *ltd_obd;struct obd_export *ltd_exp;u32 ltd_idx;struct mutex ltd_fid_mutex;
fs/lustre/lmv/lmv_obd.c +1 -1
@@ -389,7 +389,7 @@ static int lmv_add_target(struct obd_device *obd, struct obd_uuid *uuidp,if ((index < lmv->tgts_size) && lmv->tgts[index]) { tgt = lmv->tgts[index]; -CERROR ("%s: UUID %s already assigned at LOV target index %d: rc = %d\n" , +CERROR ("%s: UUID %s already assigned at LMV target index %d: rc = %d\n" , obd->obd_name,obd_uuid2str (&tgt->ltd_uuid), index, -EEXIST );mutex_unlock (&lmv->lmv_init_mutex);
fs/lustre/lov/lov_cl_internal.h +71 -5
114 lines not on this page · raw message
fs/lustre/lov/lov_dev.c +239 -37
351 lines not on this page · raw message
fs/lustre/lov/lov_ea.c +15 -5
@@ -95,7 +95,8 @@ static int lsm_lmm_verify_v1v3(struct lov_mds_md *lmm, size_t lmm_size,return -EINVAL ; } -if (lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_RAID0 ) { +if (lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_MDT && +lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_RAID0 ) {CERROR ("bad striping pattern\n" );lov_dump_lmm_common (D_WARNING , lmm);return -EINVAL ;
@@ -206,6 +207,12 @@ void lsm_free(struct lov_stripe_md *lsm) } } +/* with Data-on-MDT set maxbytes to stripe size */ +if (lsme_is_dom (lsme)) { + lov_bytes = lsme->lsme_stripe_size; + goto out_dom; + } +for (i =0 ; i < stripe_count; i++) {struct lov_tgt_desc *ltd;struct lov_oinfo *loi;
@@ -253,6 +260,7 @@ void lsm_free(struct lov_stripe_md *lsm) lov_bytes = min_stripe_maxbytes * stripe_count; +out_dom:if (maxbytes) {if (lov_bytes < min_stripe_maxbytes)/* handle overflow */ *maxbytes =MAX_LFS_FILESIZE ;
@@ -385,7 +393,8 @@ static int lsm_verify_comp_md_v1(struct lov_comp_md_v1 *lcm,unsigned int magic; stripe_count =le16_to_cpu (lmm->lmm_stripe_count); -if (stripe_count ==0 ) +if (stripe_count ==0 && +lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_MDT )return ERR_PTR (-EINVAL );/* un-instantiated lmm contains no ost id info, i.e. lov_ost_data_v1 */
@@ -474,9 +483,10 @@ static int lsm_verify_comp_md_v1(struct lov_comp_md_v1 *lcm,/* the last component hasn't been defined, or * lsm_maxbytes overflowed. */ -if (lsme->lsme_extent.e_end !=LUSTRE_EOF || - lsm->lsm_maxbytes < - (loff_t )lsme->lsme_extent.e_start) +if (!lsme_is_dom (lsme) && + (lsme->lsme_extent.e_end !=LUSTRE_EOF || + lsm->lsm_maxbytes < + (loff_t )lsme->lsme_extent.e_start)) lsm->lsm_maxbytes =MAX_LFS_FILESIZE ; } }
fs/lustre/lov/lov_internal.h +7 -0
@@ -57,6 +57,11 @@ struct lov_stripe_md_entry {struct lov_oinfo *lsme_oinfo[]; }; +static inline bool lsme_is_dom (struct lov_stripe_md_entry *lsme) +{ +return (lov_pattern (lsme->lsme_pattern) ==LOV_PATTERN_MDT ); +} +static inline void copy_lsm_entry (struct lov_stripe_md_entry *dst,struct lov_stripe_md_entry *src) {
@@ -300,6 +305,8 @@ struct lov_stripe_md *lov_unpackmd(struct lov_obd *lov, void *buf,/* lov_cl.c */ extern struct lu_device_type lov_device_type; +#define LOV_MDC_TGT_MAX 256 +/* ost_pool methods */ int lov_ost_pool_init (struct ost_pool *op,unsigned int count);int lov_ost_pool_extend (struct ost_pool *op,unsigned int min_count);
fs/lustre/lov/lov_io.c +5 -1
@@ -533,7 +533,11 @@ static int lov_io_setattr_iter_init(const struct lu_env *env,if (cl_io_is_trunc (io) && lio->lis_pos >0 ) { index =lov_lsm_entry (lsm, lio->lis_pos -1 ); -if (index >0 && !lsm_entry_inited (lsm, index)) { +/* no entry found for such offset */ +if (index <0 ) { + io->ci_result = -ENODATA ; +return io->ci_result; + }else if (!lsm_entry_inited (lsm, index)) { io->ci_need_write_intent =1 ; io->ci_result = -ENODATA ;return io->ci_result;
fs/lustre/lov/lov_obd.c +29 -10
@@ -852,6 +852,9 @@ int lov_process_config_base(struct obd_device *obd, struct lustre_cfg *lcfg,int rc =0 ;switch (cmd = lcfg->lcfg_command) { +case LCFG_ADD_MDC : +case LCFG_DEL_MDC : +break ;case LCFG_LOV_ADD_OBD :case LCFG_LOV_ADD_INA :case LCFG_LOV_DEL_OBD : {
@@ -1179,31 +1182,32 @@ static int lov_set_info_async(const struct lu_env *env, struct obd_export *exp, {struct obd_device *obddev =class_exp2obd (exp);struct lov_obd *lov = &obddev->u.lov; -u32 count; -int i, rc =0 , err;struct lov_tgt_desc *tgt; -int do_inactive =0 , no_set =0 ; +bool do_inactive = false; +bool no_set = false; +int rc =0 ; +int err; +u32 i;if (!set) { - no_set =1 ; + no_set = true; set =ptlrpc_prep_set ();if (!set)return -ENOMEM ; }lov_tgts_getref (obddev); - count = lov->desc.ld_tgt_count;if (KEY_IS (KEY_CHECKSUM )) { - do_inactive =1 ; + do_inactive = true; }else if (KEY_IS (KEY_CACHE_SET )) {LASSERT (!lov->lov_cache); lov->lov_cache = val; - do_inactive =1 ; + do_inactive = true;cl_cache_incref (lov->lov_cache); } -for (i =0 ; i < count; i++) { +for (i =0 ; i < lov->desc.ld_tgt_count; i++) { tgt = lov->lov_tgts[i];/* OST was disconnected */
@@ -1216,14 +1220,29 @@ static int lov_set_info_async(const struct lu_env *env, struct obd_export *exp, err =obd_set_info_async (env, tgt->ltd_exp, keylen, key, vallen, val, set); -if (!rc) + +if (rc ==0 ) + rc = err; + } + +/* cycle through MDC target for Data-on-MDT */ +for (i =0 ; i <LOV_MDC_TGT_MAX ; i++) { +struct obd_device *mdc; + + mdc = lov->lov_mdc_tgts[i].lmtd_mdc; +if (!mdc) +continue ; + + err =obd_set_info_async (env, mdc->obd_self_export, + keylen, key, vallen, val, set); +if (rc ==0 ) rc = err; }lov_tgts_putref (obddev);if (no_set) { err =ptlrpc_set_wait (set); -if (!rc) +if (rc ==0 ) rc = err;ptlrpc_set_destroy (set); }
fs/lustre/lov/lov_object.c +461 -235
861 lines not on this page · raw message
fs/lustre/lov/lov_offset.c +3 -0
@@ -43,6 +43,9 @@ static u64 stripe_width(struct lov_stripe_md *lsm, unsigned int index)LASSERT (index < lsm->lsm_entry_count); +if (lsme_is_dom (entry)) +return (loff_t )entry->lsme_stripe_size; +return entry->lsme_stripe_size * entry->lsme_stripe_count; }
fs/lustre/mdc/mdc_request.c +6 -1
@@ -2265,7 +2265,12 @@ static int mdc_set_info_async(const struct lu_env *env,return 0 ; } -CERROR ("Unknown key %s\n" , (char *)key); +/* TODO: these OSC-related keys are ignored for now */ +if (KEY_IS (KEY_CHECKSUM ) ||KEY_IS (KEY_CACHE_SET ) || +KEY_IS (KEY_CACHE_LRU_SHRINK ) ||KEY_IS (KEY_GRANT_SHRINK )) +return 0 ; + +CERROR ("%s: Unknown key %s\n" , exp->exp_obd->obd_name, (char *)key);return -EINVAL ; }
fs/lustre/obdclass/obd_config.c +35 -1
@@ -972,7 +972,6 @@ int class_process_config(struct lustre_cfg *lcfg) err = -EINVAL ; goto out; } -switch (lcfg->lcfg_command) {case LCFG_SETUP : { err =class_setup (obd, lcfg);
@@ -1020,6 +1019,41 @@ int class_process_config(struct lustre_cfg *lcfg) err =0 ; goto out; } +/* Process config log ADD_MDC record twice to add MDC also to LOV +* for Data-on-MDT: +* +* add 0:lustre-clilmv 1:lustre-MDT0000_UUID 2:0 3:1 +* 4:lustre-MDT0000-mdc_UUID +*/ +case LCFG_ADD_MDC : { +struct obd_device *lov_obd; +char *clilmv; + + err =obd_process_config (obd,sizeof (*lcfg), lcfg); +if (err) + goto out; + +/* make sure this is client LMV log entry */ + clilmv =strstr (lustre_cfg_string (lcfg,0 ),"clilmv" ); +if (!clilmv) + goto out; + +/* replace 'lmv' with 'lov' name to address LOV device and +* process llog record to add MDC there. +*/ + clilmv[4 ] ='o' ; + lov_obd =class_name2obd (lustre_cfg_string (lcfg,0 )); +if (!lov_obd) { + err = -ENOENT ; +CERROR ("%s: Cannot find LOV by %s name, rc = %d\n" , + obd->obd_name,lustre_cfg_string (lcfg,0 ), err); + }else { + err =obd_process_config (lov_obd,sizeof (*lcfg), lcfg); + } +/* restore 'lmv' name */ + clilmv[4 ] ='m' ; + goto out; + }default : { err =obd_process_config (obd,sizeof (*lcfg), lcfg); goto out;
fs/lustre/ptlrpc/wiretest.c +2 -2
@@ -1479,8 +1479,8 @@ void lustre_assert_wire_constants(void) (unsigned int)LOV_PATTERN_RAID0);LASSERTF (LOV_PATTERN_RAID1 ==0x00000002UL ,"found 0x%.8xUL\n" , (unsigned int )LOV_PATTERN_RAID1 ); -LASSERTF (LOV_PATTERN_FIRST ==0x00000100UL ,"found 0x%.8xUL\n" , - (unsigned int )LOV_PATTERN_FIRST ); +LASSERTF (LOV_PATTERN_MDT ==0x00000100UL ,"found 0x%.8xUL\n" , + (unsigned int )LOV_PATTERN_MDT );LASSERTF (LOV_PATTERN_CMOBD ==0x00000200UL ,"found 0x%.8xUL\n" , (unsigned int )LOV_PATTERN_CMOBD );
include/uapi/linux/lustre/lustre_user.h +1 -1
@@ -337,7 +337,7 @@ enum ll_lease_type {#define LOV_PATTERN_RAID0 0x001#define LOV_PATTERN_RAID1 0x002 -#define LOV_PATTERN_FIRST 0x100 +#define LOV_PATTERN_MDT 0x100#define LOV_PATTERN_CMOBD 0x200#define LOV_PATTERN_F_MASK 0xffff0000
Signature · 1 line
1.8.3.1
PATCH 43/151 lustre: lov: add MDT target to the LOV device
Quoted 11 lines · James Simmons
From: Mikhal Pershin <[email protected]>
MDC becomes LOV target like OSC for Data-on-MDT needs. Patch does the following:
- new composite layout entry type is added -LLT_DOMto describe Data-on-MDT striping.
- LOV process config log and checks for MDC targets organizing them separately from OSCs
- LOV operations are changed where needed to understand new layout entry type
Quoted 7 lines
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit:8b352709a66f("LU-3285 lov: add MDT target to the LOV device")Signed-off-by Mikhal Pershin <[email protected]>Reviewed-on: https://review.whamcloud.com/28010
Reviewed-by Jinshan Xiong <[email protected]>Reviewed-by Andreas Dilger <[email protected]>Signed-off-by James Simmons <[email protected]>
Hi James,
you appear to have merged (most of) my
lustre: use wait_event() in lov_subobject_kill()
patch into this. What that intentional?
NeilBrown
Quoted 1,737 lines
--- fs/lustre/include/obd.h | 8 + fs/lustre/lmv/lmv_obd.c | 2 +- fs/lustre/lov/lov_cl_internal.h | 76 +++- fs/lustre/lov/lov_dev.c | 276 +++++++++++-- fs/lustre/lov/lov_ea.c | 20 +- fs/lustre/lov/lov_internal.h | 7 + fs/lustre/lov/lov_io.c | 6 +- fs/lustre/lov/lov_obd.c | 39 +- fs/lustre/lov/lov_object.c | 696 +++++++++++++++++++++----------- fs/lustre/lov/lov_offset.c | 3 + fs/lustre/mdc/mdc_request.c | 7 +- fs/lustre/obdclass/obd_config.c | 36 +- fs/lustre/ptlrpc/wiretest.c | 4 +- include/uapi/linux/lustre/lustre_user.h | 2 +- 14 files changed, 883 insertions(+), 299 deletions(-)fs/lustre/include/obd.h +8
-0@@ -381,6 +381,11 @@ struct lov_tgt_desc { ltd_reap:1 ;/* should this target be deleted */ }; +struct lov_md_tgt_desc { +struct obd_device *lmtd_mdc; +u32 lmtd_index; +}; +struct lov_obd {struct lov_desc desc;struct lov_tgt_desc **lov_tgts;/* sparse array */ @@ -403,10 +408,13 @@ struct lov_obd {struct rw_semaphore lov_notify_lock;struct kobject *lov_tgts_kobj; +/* Data-on-MDT: MDC array */ +struct lov_md_tgt_desc *lov_mdc_tgts; };struct lmv_tgt_desc {struct obd_uuid ltd_uuid; +struct obd_device *ltd_obd;struct obd_export *ltd_exp;u32 ltd_idx;struct mutex ltd_fid_mutex;fs/lustre/lmv/lmv_obd.c +1
-1@@ -389,7 +389,7 @@ static int lmv_add_target(struct obd_device *obd, struct obd_uuid *uuidp,if ((index < lmv->tgts_size) && lmv->tgts[index]) { tgt = lmv->tgts[index]; -CERROR ("%s: UUID %s already assigned at LOV target index %d: rc = %d\n" , +CERROR ("%s: UUID %s already assigned at LMV target index %d: rc = %d\n" , obd->obd_name,obd_uuid2str (&tgt->ltd_uuid), index, -EEXIST );mutex_unlock (&lmv->lmv_init_mutex);fs/lustre/lov/lov_cl_internal.h +71
-5@@ -91,6 +91,12 @@ enum lov_device_flags {* Upper half. */ +/* Data-on-MDT array item in lov_device::ld_md_tgts[] */ +struct lovdom_device { +struct cl_device *ldm_mdc; +int ldm_idx; +}; +struct lov_device {/* * XXX Locking of lov-private data is missing. @@ -101,6 +107,13 @@ struct lov_device {u32 ld_target_nr;struct lovsub_device **ld_target;u32 ld_flags; + +/* Data-on-MDT devices */ +u32 ld_md_tgts_nr; +struct lovdom_device *ld_md_tgts; +struct obd_device *ld_lmv; +/* LU site for subdevices */ +struct lu_site ld_site; };/** @@ -129,6 +142,34 @@ static inline char *llt2str(enum lov_layout_type llt)return "" ; } +/** +* Return lov_layout_entry_type associated with a given composite layout +* entry. +*/ +static inline u32 lov_entry_type (struct lov_stripe_md_entry *lsme) +{ +if ((lov_pattern (lsme->lsme_pattern) ==LOV_PATTERN_RAID0 ) || + (lov_pattern (lsme->lsme_pattern) ==LOV_PATTERN_MDT )) +return lov_pattern (lsme->lsme_pattern); +return 0 ; +} + +struct lov_layout_entry ; +struct lov_object ; +struct lov_lock_sub ; + +struct lov_comp_layout_entry_ops { +int (*lco_init)(const struct lu_env *env,struct lov_device *dev, +struct lov_object *lov,unsigned int index, +const struct cl_object_conf *conf, +struct lov_layout_entry *lle); +void (*lco_fini)(const struct lu_env *env, +struct lov_layout_entry *lle); +int (*lco_getattr)(const struct lu_env *env,struct lov_object *obj, +unsigned int index,struct lov_layout_entry *lle, +struct cl_attr **attr); +}; +struct lov_layout_raid0 {unsigned int lo_nr;/** @@ -165,6 +206,25 @@ struct lov_layout_raid0 {struct cl_attr lo_attr; }; +struct lov_layout_dom { +/* keep this always at first place so DOM layout entry +* can be addressed also as RAID0 after initialization. +*/ +struct lov_layout_raid0 lo_dom_r0; +struct lovsub_object *lo_dom; +struct lov_oinfo *lo_loi; +}; + +struct lov_layout_entry { +u32 lle_type; +struct lu_extent lle_extent; +struct lov_comp_layout_entry_ops *lle_comp_ops; +union { +struct lov_layout_raid0 lle_raid0; +struct lov_layout_dom lle_dom; + }; +}; +/** * lov-specific file state. * @@ -220,13 +280,10 @@ struct lov_object { } released;struct lov_layout_composite {/** -* Current valid entry count of lo_entries. +* Current valid entry count of entries. */ unsigned int lo_entry_count; -struct lov_layout_entry { -struct lu_extent lle_extent; -struct lov_layout_raid0 lle_raid0; - } *lo_entries; +struct lov_layout_entry *lo_entries; } composite; } u;/** @@ -633,6 +690,15 @@ static inline struct lov_thread_info *lov_env_info(const struct lu_env *env)return info; } +static inline struct lov_layout_entry *lov_entry (struct lov_object *lov,int i) +{ +LASSERT (lov->lo_type ==LLT_COMP ); +LASSERTF (i < lov->u.composite.lo_entry_count, +"entry %d entry_count %d" , i, lov->u.composite.lo_entry_count); + +return &lov->u.composite.lo_entries[i]; +} +static inline struct lov_layout_raid0 *lov_r0 (struct lov_object *lov,int i) {LASSERT (lov->lo_type ==LLT_COMP );fs/lustre/lov/lov_dev.c +239
-37@@ -146,23 +146,55 @@ struct lu_context_key lov_session_key = {/* type constructor/destructor: lov_type_{init,fini,start,stop}() */ LU_TYPE_INIT_FINI (lov, &lov_key, &lov_session_key); + +static int lov_mdc_dev_init (const struct lu_env *env,struct lov_device *ld, +struct lu_device *mdc_dev,u32 idx,u32 nr) +{ +struct cl_device *cl; + + cl =cl_type_setup (env, &ld->ld_site, &lovsub_device_type, + mdc_dev); +if (IS_ERR (cl)) +return PTR_ERR (cl); + + ld->ld_md_tgts[nr].ldm_mdc = cl; + ld->ld_md_tgts[nr].ldm_idx = idx; +return 0 ; +} +static struct lu_device *lov_device_fini (const struct lu_env *env,struct lu_device *d) { -int i;struct lov_device *ld =lu2lov_dev (d); +int i;LASSERT (ld->ld_lov); -if (!ld->ld_target) -return NULL ; - lov_foreach_target(ld, i) { -struct lovsub_device *lsd; +if (ld->ld_lmv) { +class_decref (ld->ld_lmv,"lov" , d); + ld->ld_lmv =NULL ; + } + +if (ld->ld_md_tgts) { +for (i =0 ; i < ld->ld_md_tgts_nr; i++) { +if (!ld->ld_md_tgts[i].ldm_mdc) +continue ; - lsd = ld->ld_target[i]; -if (lsd) { -cl_stack_fini (env,lovsub2cl_dev (lsd)); - ld->ld_target[i] =NULL ; +cl_stack_fini (env, ld->ld_md_tgts[i].ldm_mdc); + ld->ld_md_tgts[i].ldm_mdc =NULL ; + ld->ld_lov->lov_mdc_tgts[i].lmtd_mdc =NULL ; + } + } + +if (ld->ld_target) { + lov_foreach_target(ld, i) { +struct lovsub_device *lsd; + + lsd = ld->ld_target[i]; +if (lsd) { +cl_stack_fini (env,lovsub2cl_dev (lsd)); + ld->ld_target[i] =NULL ; + } } }return NULL ;@@ -175,9 +207,28 @@ static int lov_device_init(const struct lu_env *env, struct lu_device *d,int i;int rc =0 ; -LASSERT (d->ld_site); +/* check all added already MDC subdevices and initialize them */ +for (i =0 ; i < ld->ld_md_tgts_nr; i++) { +struct obd_device *mdc; +u32 idx; + + mdc = ld->ld_lov->lov_mdc_tgts[i].lmtd_mdc; + idx = ld->ld_lov->lov_mdc_tgts[i].lmtd_index; + +if (!mdc) +continue ; + + rc =lov_mdc_dev_init (env, ld, mdc->obd_lu_dev, idx, i); +if (rc) { +CERROR ("%s: failed to add MDC %s as target: rc = %d\n" , + d->ld_obd->obd_name, +obd_uuid2str (&mdc->obd_uuid), rc); + goto out_err; + } + } +if (!ld->ld_target) -return rc; +return 0 ; lov_foreach_target(ld, i) {struct lovsub_device *lsd;@@ -188,21 +239,21 @@ static int lov_device_init(const struct lu_env *env, struct lu_device *d,if (!desc)continue ; - cl =cl_type_setup (env, d->ld_site, &lovsub_device_type, + cl =cl_type_setup (env, &ld->ld_site, &lovsub_device_type, desc->ltd_obd->obd_lu_dev);if (IS_ERR (cl)) { rc =PTR_ERR (cl); -break ; + goto out_err; } + lsd =cl2lovsub_dev (cl); ld->ld_target[i] = lsd; } + ld->ld_flags |=LOV_DEV_INITIALIZED ; +return 0 ; -if (rc) -lov_device_fini (env, d); -else - ld->ld_flags |=LOV_DEV_INITIALIZED ; - +out_err: +lu_device_fini (d);return rc; }@@ -211,8 +262,17 @@ static struct lu_device *lov_device_free(const struct lu_env *env, {struct lov_device *ld =lu2lov_dev (d); +lu_site_fini (&ld->ld_site); +cl_device_fini (lu2cl_dev (d));kfree (ld->ld_target); + ld->ld_target =NULL ; +kfree (ld->ld_md_tgts); + ld->ld_md_tgts =NULL ; +/* free array of MDCs */ +kfree (ld->ld_lov->lov_mdc_tgts); + ld->ld_lov->lov_mdc_tgts =NULL ; +kfree (ld);return NULL ; }@@ -277,9 +337,7 @@ static int lov_cl_add_target(const struct lu_env *env, struct lu_device *dev, rc =lov_expand_targets (env, ld);if (rc ==0 && ld->ld_flags &LOV_DEV_INITIALIZED ) { -LASSERT (dev->ld_site); - - cl =cl_type_setup (env, dev->ld_site, &lovsub_device_type, + cl =cl_type_setup (env, &ld->ld_site, &lovsub_device_type, tgt->ltd_obd->obd_lu_dev);if (!IS_ERR (cl)) { lsd =cl2lovsub_dev (cl);@@ -297,6 +355,84 @@ static int lov_cl_add_target(const struct lu_env *env, struct lu_device *dev,return rc; } +/** +* Add new MDC target device in LOV. +* +* This function is part of the configuration log processing. It adds new MDC +* device to the MDC device array indexed by their indexes. +* +* @env execution environment +* @d LU device of LOV device +* @mdc MDC device to add +* @idx MDC device index +* +* Return: 0 if successful +* negative value on error +*/ +static int lov_add_mdc_target (const struct lu_env *env,struct lu_device *d, +struct obd_device *mdc,u32 idx) +{ +struct lov_device *ld =lu2lov_dev (d); +struct obd_device *lov_obd = d->ld_obd; +struct obd_device *lmv_obd; +int next; +int rc =0 ; + +LASSERT (mdc); +if (ld->ld_md_tgts_nr ==LOV_MDC_TGT_MAX ) { +/* If the maximum value of LOV_MDC_TGT_MAX will become too +* small then all MD target handling must be rewritten in LOD +* manner, check lod_add_device() and related functionality. +*/ +CERROR ("%s: cannot serve more than %d MDC devices\n" , + lov_obd->obd_name,LOV_MDC_TGT_MAX ); +return -ERANGE ; + } + +/* grab FLD from lmv, do that here, when first MDC is added +* to be sure LMV is set up and can be found +*/ +if (!ld->ld_lmv) { + next =0 ; +while ((lmv_obd =class_devices_in_group (&lov_obd->obd_uuid, + &next)) !=NULL ) { +if ((strncmp (lmv_obd->obd_type->typ_name, +LUSTRE_LMV_NAME , +strlen (LUSTRE_LMV_NAME )) ==0 )) +break ; + } +if (!lmv_obd) { +CERROR ("%s: cannot find LMV OBD by UUID (%s)\n" , + lov_obd->obd_name, +obd_uuid2str (&lmv_obd->obd_uuid)); +return -ENODEV ; + } +spin_lock (&lmv_obd->obd_dev_lock); +class_incref (lmv_obd,"lov" , ld); +spin_unlock (&lmv_obd->obd_dev_lock); + ld->ld_lmv = lmv_obd; + } + +LASSERT (!lov_obd->u.lov.lov_mdc_tgts[ld->ld_md_tgts_nr].lmtd_mdc); + +if (ld->ld_flags &LOV_DEV_INITIALIZED ) { + rc =lov_mdc_dev_init (env, ld, mdc->obd_lu_dev, idx, + ld->ld_md_tgts_nr); +if (rc) { +CERROR ("%s: failed to add MDC %s as target: rc = %d\n" , + lov_obd->obd_name,obd_uuid2str (&mdc->obd_uuid), + rc); +return rc; + } + } + + lov_obd->u.lov.lov_mdc_tgts[ld->ld_md_tgts_nr].lmtd_mdc = mdc; + lov_obd->u.lov.lov_mdc_tgts[ld->ld_md_tgts_nr].lmtd_index = idx; + ld->ld_md_tgts_nr++; + +return rc; +} +static int lov_process_config (const struct lu_env *env,struct lu_device *d,struct lustre_cfg *cfg) {@@ -309,23 +445,52 @@ static int lov_process_config(const struct lu_env *env,lov_tgts_getref (obd); cmd = cfg->lcfg_command; + rc =lov_process_config_base (d->ld_obd, cfg, &index, &gen); -if (rc ==0 ) { -switch (cmd) { -case LCFG_LOV_ADD_OBD : -case LCFG_LOV_ADD_INA : - rc =lov_cl_add_target (env, d, index); -if (rc !=0 ) -lov_del_target (d->ld_obd, index,NULL ,0 ); -break ; -case LCFG_LOV_DEL_OBD : -lov_cl_del_target (env, d, index); -break ; +if (rc <0 ) + goto out; + +switch (cmd) { +case LCFG_LOV_ADD_OBD : +case LCFG_LOV_ADD_INA : + rc =lov_cl_add_target (env, d, index); +if (rc !=0 ) +lov_del_target (d->ld_obd, index,NULL ,0 ); +break ; +case LCFG_LOV_DEL_OBD : +lov_cl_del_target (env, d, index); +break ; +case LCFG_ADD_MDC : + { +struct obd_device *mdc; +struct obd_uuid tgt_uuid; + +/* modify_mdc_tgts add 0:lustre-clilmv 1:lustre-MDT0000_UUID +* 2:0 3:1 4:lustre-MDT0000-mdc_UUID +*/ +if (LUSTRE_CFG_BUFLEN (cfg,1 ) >sizeof (tgt_uuid.uuid)) { + rc = -EINVAL ; + goto out; } - } -lov_tgts_putref (obd); +obd_str2uuid (&tgt_uuid,lustre_cfg_buf (cfg,1 )); +if (sscanf (lustre_cfg_buf (cfg,2 ),"%d" , &index) !=1 ) { + rc = -EINVAL ; + goto out; + } + mdc =class_find_client_obd (&tgt_uuid,LUSTRE_MDC_NAME , + &obd->obd_uuid); +if (!mdc) { + rc = -ENODEV ; + goto out; + } + rc =lov_add_mdc_target (env, d, mdc, index); +break ; + } + } +out: +lov_tgts_putref (obd);return rc; }@@ -355,13 +520,50 @@ static struct lu_device *lov_device_alloc(const struct lu_env *env, obd =class_name2obd (lustre_cfg_string (cfg,0 ));LASSERT (obd); rc =lov_setup (obd, cfg); -if (rc) { -lov_device_free (env, d); -return ERR_PTR (rc); +if (rc) + goto out; + +/* Alloc MDC devices array */ +/* XXX: need dynamic allocation at some moment */ + ld->ld_md_tgts =kcalloc (LOV_MDC_TGT_MAX ,sizeof (*ld->ld_md_tgts), +GFP_NOFS ); +if (!ld->ld_md_tgts) { + rc = -ENOMEM ; + goto out; } + ld->ld_md_tgts_nr =0 ; ld->ld_lov = &obd->u.lov; + ld->ld_lov->lov_mdc_tgts = +kcalloc (LOV_MDC_TGT_MAX , +sizeof (*ld->ld_lov->lov_mdc_tgts), +GFP_NOFS ); +if (!ld->ld_lov->lov_mdc_tgts) { + rc = -ENOMEM ; + goto out_md_tgts; + } + + rc =lu_site_init (&ld->ld_site, d); +if (rc !=0 ) + goto out_mdc_tgts; + + rc =lu_site_init_finish (&ld->ld_site); +if (rc !=0 ) + goto out_site; +return d; +out_site: +lu_site_fini (&ld->ld_site); +out_mdc_tgts: +kfree (ld->ld_lov->lov_mdc_tgts); + ld->ld_lov->lov_mdc_tgts =NULL ; +out_md_tgts: +kfree (ld->ld_md_tgts); + ld->ld_md_tgts =NULL ; +out: +kfree (ld); + +return ERR_PTR (rc); }static const struct lu_device_type_operations lov_device_type_ops = {fs/lustre/lov/lov_ea.c +15
-5@@ -95,7 +95,8 @@ static int lsm_lmm_verify_v1v3(struct lov_mds_md *lmm, size_t lmm_size,return -EINVAL ; } -if (lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_RAID0 ) { +if (lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_MDT && +lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_RAID0 ) {CERROR ("bad striping pattern\n" );lov_dump_lmm_common (D_WARNING , lmm);return -EINVAL ;@@ -206,6 +207,12 @@ void lsm_free(struct lov_stripe_md *lsm) } } +/* with Data-on-MDT set maxbytes to stripe size */ +if (lsme_is_dom (lsme)) { + lov_bytes = lsme->lsme_stripe_size; + goto out_dom; + } +for (i =0 ; i < stripe_count; i++) {struct lov_tgt_desc *ltd;struct lov_oinfo *loi;@@ -253,6 +260,7 @@ void lsm_free(struct lov_stripe_md *lsm) lov_bytes = min_stripe_maxbytes * stripe_count; +out_dom:if (maxbytes) {if (lov_bytes < min_stripe_maxbytes)/* handle overflow */ *maxbytes =MAX_LFS_FILESIZE ;@@ -385,7 +393,8 @@ static int lsm_verify_comp_md_v1(struct lov_comp_md_v1 *lcm,unsigned int magic; stripe_count =le16_to_cpu (lmm->lmm_stripe_count); -if (stripe_count ==0 ) +if (stripe_count ==0 && +lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_MDT )return ERR_PTR (-EINVAL );/* un-instantiated lmm contains no ost id info, i.e. lov_ost_data_v1 */ @@ -474,9 +483,10 @@ static int lsm_verify_comp_md_v1(struct lov_comp_md_v1 *lcm,/* the last component hasn't been defined, or * lsm_maxbytes overflowed. */ -if (lsme->lsme_extent.e_end !=LUSTRE_EOF || - lsm->lsm_maxbytes < - (loff_t )lsme->lsme_extent.e_start) +if (!lsme_is_dom (lsme) && + (lsme->lsme_extent.e_end !=LUSTRE_EOF || + lsm->lsm_maxbytes < + (loff_t )lsme->lsme_extent.e_start)) lsm->lsm_maxbytes =MAX_LFS_FILESIZE ; } }fs/lustre/lov/lov_internal.h +7
-0@@ -57,6 +57,11 @@ struct lov_stripe_md_entry {struct lov_oinfo *lsme_oinfo[]; }; +static inline bool lsme_is_dom (struct lov_stripe_md_entry *lsme) +{ +return (lov_pattern (lsme->lsme_pattern) ==LOV_PATTERN_MDT ); +} +static inline void copy_lsm_entry (struct lov_stripe_md_entry *dst,struct lov_stripe_md_entry *src) {@@ -300,6 +305,8 @@ struct lov_stripe_md *lov_unpackmd(struct lov_obd *lov, void *buf,/* lov_cl.c */ extern struct lu_device_type lov_device_type; +#define LOV_MDC_TGT_MAX 256 +/* ost_pool methods */ int lov_ost_pool_init (struct ost_pool *op,unsigned int count);int lov_ost_pool_extend (struct ost_pool *op,unsigned int min_count);fs/lustre/lov/lov_io.c +5
-1@@ -533,7 +533,11 @@ static int lov_io_setattr_iter_init(const struct lu_env *env,if (cl_io_is_trunc (io) && lio->lis_pos >0 ) { index =lov_lsm_entry (lsm, lio->lis_pos -1 ); -if (index >0 && !lsm_entry_inited (lsm, index)) { +/* no entry found for such offset */ +if (index <0 ) { + io->ci_result = -ENODATA ; +return io->ci_result; + }else if (!lsm_entry_inited (lsm, index)) { io->ci_need_write_intent =1 ; io->ci_result = -ENODATA ;return io->ci_result;fs/lustre/lov/lov_obd.c +29
-10@@ -852,6 +852,9 @@ int lov_process_config_base(struct obd_device *obd, struct lustre_cfg *lcfg,int rc =0 ;switch (cmd = lcfg->lcfg_command) { +case LCFG_ADD_MDC : +case LCFG_DEL_MDC : +break ;case LCFG_LOV_ADD_OBD :case LCFG_LOV_ADD_INA :case LCFG_LOV_DEL_OBD : {@@ -1179,31 +1182,32 @@ static int lov_set_info_async(const struct lu_env *env, struct obd_export *exp, {struct obd_device *obddev =class_exp2obd (exp);struct lov_obd *lov = &obddev->u.lov; -u32 count; -int i, rc =0 , err;struct lov_tgt_desc *tgt; -int do_inactive =0 , no_set =0 ; +bool do_inactive = false; +bool no_set = false; +int rc =0 ; +int err; +u32 i;if (!set) { - no_set =1 ; + no_set = true; set =ptlrpc_prep_set ();if (!set)return -ENOMEM ; }lov_tgts_getref (obddev); - count = lov->desc.ld_tgt_count;if (KEY_IS (KEY_CHECKSUM )) { - do_inactive =1 ; + do_inactive = true; }else if (KEY_IS (KEY_CACHE_SET )) {LASSERT (!lov->lov_cache); lov->lov_cache = val; - do_inactive =1 ; + do_inactive = true;cl_cache_incref (lov->lov_cache); } -for (i =0 ; i < count; i++) { +for (i =0 ; i < lov->desc.ld_tgt_count; i++) { tgt = lov->lov_tgts[i];/* OST was disconnected */ @@ -1216,14 +1220,29 @@ static int lov_set_info_async(const struct lu_env *env, struct obd_export *exp, err =obd_set_info_async (env, tgt->ltd_exp, keylen, key, vallen, val, set); -if (!rc) + +if (rc ==0 ) + rc = err; + } + +/* cycle through MDC target for Data-on-MDT */ +for (i =0 ; i <LOV_MDC_TGT_MAX ; i++) { +struct obd_device *mdc; + + mdc = lov->lov_mdc_tgts[i].lmtd_mdc; +if (!mdc) +continue ; + + err =obd_set_info_async (env, mdc->obd_self_export, + keylen, key, vallen, val, set); +if (rc ==0 ) rc = err; }lov_tgts_putref (obddev);if (no_set) { err =ptlrpc_set_wait (set); -if (!rc) +if (rc ==0 ) rc = err;ptlrpc_set_destroy (set); }fs/lustre/lov/lov_object.c +461
-235@@ -90,13 +90,6 @@ static void lov_lsm_put(struct lov_stripe_md *lsm)* Lov object layout operations. * */ -static int lov_init_empty (const struct lu_env *env,struct lov_device *dev, -struct lov_object *lov,struct lov_stripe_md *lsm, -const struct cl_object_conf *conf, -union lov_layout_state *state) -{ -return 0 ; -} static struct cl_object *lov_sub_find(const struct lu_env *env, struct cl_device *dev,@@ -110,9 +103,25 @@ static struct cl_object *lov_sub_find(const struct lu_env *env,return lu2cl (o); } +static int lov_page_slice_fixup (struct lov_object *lov, +struct cl_object *stripe) +{ +struct cl_object_header *hdr =cl_object_header (&lov->lo_cl); +struct cl_object *o; + +if (!stripe) +return hdr->coh_page_bufsize - lov->lo_cl.co_slice_off - +cfs_size_round (sizeof (struct lov_page )); + +cl_object_for_each (o, stripe) + o->co_slice_off += hdr->coh_page_bufsize; + +return cl_object_header (stripe)->coh_page_bufsize; +} +static int lov_init_sub (const struct lu_env *env,struct lov_object *lov, -struct cl_object *subobj,struct lov_layout_raid0 *r0, -struct lov_oinfo *oinfo,int idx) +struct cl_object *subobj,struct lov_oinfo *oinfo, +int idx) {int stripe =lov_comp_stripe (idx);int entry =lov_comp_entry (idx);@@ -146,13 +155,14 @@ static int lov_init_sub(const struct lu_env *env, struct lov_object *lov,spin_lock (&subhdr->coh_attr_guard); parent = subhdr->coh_parent;if (!parent) { +struct lovsub_object *lso =cl2lovsub (subobj); + subhdr->coh_parent = hdr;spin_unlock (&subhdr->coh_attr_guard); subhdr->coh_nesting = hdr->coh_nesting +1 ;lu_object_ref_add (&subobj->co_lu,"lov-parent" , lov); - r0->lo_sub[stripe] =cl2lovsub (subobj); - r0->lo_sub[stripe]->lso_super = lov; - r0->lo_sub[stripe]->lso_index = idx; + lso->lso_super = lov; + lso->lso_index = idx; result =0 ; }else {struct lu_object *old_obj;@@ -183,33 +193,19 @@ static int lov_init_sub(const struct lu_env *env, struct lov_object *lov,return result; } -static int lov_page_slice_fixup (struct lov_object *lov, -struct cl_object *stripe) -{ -struct cl_object_header *hdr =cl_object_header (&lov->lo_cl); -struct cl_object *o; - -if (!stripe) -return hdr->coh_page_bufsize - lov->lo_cl.co_slice_off - -cfs_size_round (sizeof (struct lov_page )); - -cl_object_for_each (o, stripe) - o->co_slice_off += hdr->coh_page_bufsize; - -return cl_object_header (stripe)->coh_page_bufsize; -} -static int lov_init_raid0 (const struct lu_env *env,struct lov_device *dev, -struct lov_object *lov,int index, -struct lov_layout_raid0 *r0) +struct lov_object *lov,unsigned int index, +const struct cl_object_conf *conf, +struct lov_layout_entry *lle) {struct lov_stripe_md_entry *lse =lov_lse (lov, index); +struct lov_layout_raid0 *r0 = &lle->lle_raid0;struct lov_thread_info *lti =lov_env_info (env);struct cl_object_conf *subconf = <i->lti_stripe_conf;struct lu_fid *ofid = <i->lti_fid;struct cl_object *stripe;int result; -int psz; +int psz, sz;int i;spin_lock_init (&r0->lo_sub_lock);@@ -261,7 +257,7 @@ static int lov_init_raid0(const struct lu_env *env, struct lov_device *dev, goto out; } - result =lov_init_sub (env, lov, stripe, r0, oinfo, + result =lov_init_sub (env, lov, stripe, oinfo,lov_comp_index (index, i));if (result == -EAGAIN ) {/* try again */ --i;@@ -270,8 +266,9 @@ static int lov_init_raid0(const struct lu_env *env, struct lov_device *dev, }if (result ==0 ) { -int sz =lov_page_slice_fixup (lov, stripe); + r0->lo_sub[i] =cl2lovsub (stripe); + sz =lov_page_slice_fixup (lov, stripe);LASSERT (ergo (psz >0 , psz == sz)); psz = sz; }@@ -282,12 +279,333 @@ static int lov_init_raid0(const struct lu_env *env, struct lov_device *dev,return result; } +static void lov_subobject_kill (const struct lu_env *env,struct lov_object *lov, +struct lov_layout_raid0 *r0, +struct lovsub_object *los,int idx) +{ +struct cl_object *sub; +struct lu_site *site; +wait_queue_head_t *wq; + +LASSERT (r0->lo_sub[idx] == los); + + sub =lovsub2cl (los); + site = sub->co_lu.lo_dev->ld_site; + wq =lu_site_wq_from_fid (site, &sub->co_lu.lo_header->loh_fid); + +cl_object_kill (env, sub); +/* release a reference to the sub-object and ... */ +lu_object_ref_del (&sub->co_lu,"lov-parent" , lov); +cl_object_put (env, sub); + +/* ... wait until it is actually destroyed---sub-object clears its +* ->lo_sub[] slot in lovsub_object_free() +*/ +wait_event (*wq, r0->lo_sub[idx] != los); +LASSERT (!r0->lo_sub[idx]); +} + +static void lov_delete_raid0 (const struct lu_env *env,struct lov_object *lov, +struct lov_layout_entry *lle) +{ +struct lov_layout_raid0 *r0 = &lle->lle_raid0; + +if (r0->lo_sub) { +int i; + +for (i =0 ; i < r0->lo_nr; ++i) { +struct lovsub_object *los = r0->lo_sub[i]; + +if (los) { +cl_object_prune (env, &los->lso_cl); +/* +* If top-level object is to be evicted from +* the cache, so are its sub-objects. +*/ +lov_subobject_kill (env, lov, r0, los, i); + } + } + } +} + +static void lov_fini_raid0 (const struct lu_env *env, +struct lov_layout_entry *lle) +{ +struct lov_layout_raid0 *r0 = &lle->lle_raid0; + +if (r0->lo_sub) { +kvfree (r0->lo_sub); + r0->lo_sub =NULL ; + } +} + +static int lov_print_raid0 (const struct lu_env *env,void *cookie, +lu_printer_t p,const struct lov_layout_entry *lle) +{ +const struct lov_layout_raid0 *r0 = &lle->lle_raid0; +int i; + +for (i =0 ; i < r0->lo_nr; ++i) { +struct lu_object *sub; + +if (r0->lo_sub[i]) { + sub =lovsub2lu (r0->lo_sub[i]); +lu_object_print (env, cookie, p, sub); + }else { + (*p)(env, cookie,"sub %d absent\n" , i); + } + } +return 0 ; +} + +static int lov_attr_get_raid0 (const struct lu_env *env,struct lov_object *lov, +unsigned int index,struct lov_layout_entry *lle, +struct cl_attr **lov_attr) +{ +struct lov_layout_raid0 *r0 = &lle->lle_raid0; +struct lov_stripe_md *lsm = lov->lo_lsm; +struct ost_lvb *lvb = &lov_env_info (env)->lti_lvb; +struct cl_attr *attr = &r0->lo_attr; +u64 kms =0 ; +int result =0 ; + +if (r0->lo_attr_valid) { + *lov_attr = attr; +return 0 ; + } + +memset (lvb,0 ,sizeof (*lvb)); + +/* XXX: timestamps can be negative by sanity:test_39m, +* how can it be? +*/ + lvb->lvb_atime =LLONG_MIN ; + lvb->lvb_ctime =LLONG_MIN ; + lvb->lvb_mtime =LLONG_MIN ; + +/* +* XXX that should be replaced with a loop over sub-objects, +* doing cl_object_attr_get() on them. But for now, let's +* reuse old lov code. +*/ + +/* +* XXX take lsm spin-lock to keep lov_merge_lvb_kms() +* happy. It's not needed, because new code uses +* ->coh_attr_guard spin-lock to protect consistency of +* sub-object attributes. +*/ +lov_stripe_lock (lsm); + result =lov_merge_lvb_kms (lsm, index, lvb, &kms); +lov_stripe_unlock (lsm); +if (result ==0 ) { +cl_lvb2attr (attr, lvb); + attr->cat_kms = kms; + r0->lo_attr_valid =1 ; + *lov_attr = attr; + } + +return result; +} + +static struct lov_comp_layout_entry_ops raid0_ops = { + .lco_init = lov_init_raid0, + .lco_fini = lov_fini_raid0, + .lco_getattr = lov_attr_get_raid0, +}; + +static int lov_attr_get_dom (const struct lu_env *env,struct lov_object *lov, +unsigned int index,struct lov_layout_entry *lle, +struct cl_attr **lov_attr) +{ +struct lov_layout_dom *dom = &lle->lle_dom; +struct lov_oinfo *loi = dom->lo_loi; +struct cl_attr *attr = &dom->lo_dom_r0.lo_attr; + +if (dom->lo_dom_r0.lo_attr_valid) { + *lov_attr = attr; +return 0 ; + } + +if (OST_LVB_IS_ERR (loi->loi_lvb.lvb_blocks)) +return OST_LVB_GET_ERR (loi->loi_lvb.lvb_blocks); + +cl_lvb2attr (attr, &loi->loi_lvb); + attr->cat_kms = attr->cat_size > loi->loi_kms ? attr->cat_size : + loi->loi_kms; + dom->lo_dom_r0.lo_attr_valid =1 ; + *lov_attr = attr; + +return 0 ; +} + +/** +* Lookup FLD to get MDS index of the given DOM object FID. +* +* @ld LOV device +* @fid FID to lookup +* @nr index in MDC array to return back +* +* Return: 0 and @mds filled with MDS index if successful +* negative value on error +*/ +static int lov_fld_lookup (struct lov_device *ld,const struct lu_fid *fid, +u32 *nr) +{ +u32 mds_idx; +int i, rc; + + rc =fld_client_lookup (&ld->ld_lmv->u.lmv.lmv_fld,fid_seq (fid), + &mds_idx,LU_SEQ_RANGE_MDT ,NULL ); +if (rc) { +CERROR ("%s: error while looking for mds number. Seq %#llx, err = %d\n" , +lu_dev_name (cl2lu_dev (&ld->ld_cl)),fid_seq (fid), rc); +return rc; + } + +CDEBUG (D_INODE ,"FLD lookup got mds #%x for fid=" DFID "\n" , + mds_idx,PFID (fid)); + +/* find proper MDC device in the array */ +for (i =0 ; i < ld->ld_md_tgts_nr; i++) { +if (ld->ld_md_tgts[i].ldm_mdc && + ld->ld_md_tgts[i].ldm_idx == mds_idx) +break ; + } + +if (i == ld->ld_md_tgts_nr) { +CERROR ("%s: cannot find corresponding MDC device for mds #%x for fid=" DFID "\n" , +lu_dev_name (cl2lu_dev (&ld->ld_cl)), mds_idx,PFID (fid)); + rc = -EINVAL ; + }else { + *nr = i; + } +return rc; +} + +/** +* Implementation of lov_comp_layout_entry_ops::lco_init for DOM object. +* +* Init the DOM object for the first time. It prepares also RAID0 entry +* for it to use in common methods with ordinary RAID0 layout entries. +* +* @env execution environment +* @dev LOV device +* @lov LOV object +* @index Composite layout entry index in LSM +* @lle Composite LOV layout entry +*/ +static int lov_init_dom (const struct lu_env *env,struct lov_device *dev, +struct lov_object *lov,unsigned int index, +const struct cl_object_conf *conf, +struct lov_layout_entry *lle) +{ +struct lov_thread_info *lti =lov_env_info (env); +struct lov_stripe_md_entry *lsme =lov_lse (lov, index); +struct cl_object *clo; +struct lu_object *o =lov2lu (lov); +const struct lu_fid *fid =lu_object_fid (o); +struct cl_device *mdcdev; +struct lov_oinfo *loi =NULL ; +struct cl_object_conf *sconf = <i->lti_stripe_conf; +struct inode *inode = conf->coc_inode; +u32 idx =0 ; +int rc; + +LASSERT (index ==0 ); + +/* find proper MDS device */ + rc =lov_fld_lookup (dev, fid, &idx); +if (rc) +return rc; + +LASSERTF (dev->ld_md_tgts[idx].ldm_mdc, +"LOV md target[%u] is NULL\n" , idx); + +/* check lsm is DOM, more checks are needed */ +LASSERT (lsme->lsme_stripe_count ==0 ); + +/* +* Create lower cl_objects. +*/ + mdcdev = dev->ld_md_tgts[idx].ldm_mdc; + +LASSERTF (mdcdev,"non-initialized mdc subdev\n" ); + +/* DoM object has no oinfo in LSM entry, create it exclusively */ + loi =kmem_cache_zalloc (lov_oinfo_slab,GFP_NOFS ); +if (!loi) +return -ENOMEM ; + +fid_to_ostid (lu_object_fid (lov2lu (lov)), &loi->loi_oi); +/* Initialize lvb structure */ + loi->loi_lvb.lvb_mtime = inode->i_mtime.tv_sec; + loi->loi_lvb.lvb_atime = inode->i_atime.tv_sec; + loi->loi_lvb.lvb_ctime = inode->i_ctime.tv_sec; + loi->loi_lvb.lvb_blocks = inode->i_blocks; + loi->loi_lvb.lvb_size =i_size_read (inode); +if (loi->loi_lvb.lvb_size > lsme->lsme_stripe_size) + loi->loi_lvb.lvb_size = lsme->lsme_stripe_size; +loi_kms_set (loi, loi->loi_lvb.lvb_size); + + sconf->u.coc_oinfo = loi; +again: + clo =lov_sub_find (env, mdcdev, fid, sconf); +if (IS_ERR (clo)) { + rc =PTR_ERR (clo); + goto out; + } + + rc =lov_init_sub (env, lov, clo, loi,lov_comp_index (index,0 )); +if (rc == -EAGAIN )/* try again */ + goto again; +else if (rc !=0 ) + goto out; + + lle->lle_dom.lo_dom =cl2lovsub (clo); +spin_lock_init (&lle->lle_dom.lo_dom_r0.lo_sub_lock); + lle->lle_dom.lo_dom_r0.lo_nr =1 ; + lle->lle_dom.lo_dom_r0.lo_sub = &lle->lle_dom.lo_dom; + lle->lle_dom.lo_loi = loi; + + rc =lov_page_slice_fixup (lov, clo); +return rc; + +out: +kmem_cache_free (lov_oinfo_slab, loi); +return rc; +} + +/** +* Implementation of lov_layout_operations::llo_fini for DOM object. +* +* Finish the DOM object and free related memory. +* +* @env execution environment +* @lov LOV object +* @state LOV layout state +*/ +static void lov_fini_dom (const struct lu_env *env, +struct lov_layout_entry *lle) +{ +if (lle->lle_dom.lo_dom) + lle->lle_dom.lo_dom =NULL ; +kmem_cache_free (lov_oinfo_slab, lle->lle_dom.lo_loi); +} + +static struct lov_comp_layout_entry_ops dom_ops = { + .lco_init = lov_init_dom, + .lco_fini = lov_fini_dom, + .lco_getattr = lov_attr_get_dom, +}; +static int lov_init_composite (const struct lu_env *env,struct lov_device *dev,struct lov_object *lov,struct lov_stripe_md *lsm,const struct cl_object_conf *conf,union lov_layout_state *state) {struct lov_layout_composite *comp = &state->composite; +struct lov_layout_entry *lle;unsigned int entry_count;unsigned int psz =0 ;int result =0 ;@@ -306,24 +624,45 @@ static int lov_init_composite(const struct lu_env *env, struct lov_device *dev,if (!comp->lo_entries)return -ENOMEM ; +/* Initiate all entry types and extents data at first */ for (i =0 ; i < entry_count; i++) { -struct lov_layout_entry *le = &comp->lo_entries[i]; + lle = &comp->lo_entries[i]; - le->lle_extent = lsm->lsm_entries[i]->lsme_extent; + lle->lle_type =lov_entry_type (lsm->lsm_entries[i]); +switch (lle->lle_type) { +case LOV_PATTERN_RAID0 : + lle->lle_comp_ops = &raid0_ops; +break ; +case LOV_PATTERN_MDT : + lle->lle_comp_ops = &dom_ops; +break ; +default : +CERROR ("%s: unknown composite layout entry type %i\n" , +lov2obd (dev->ld_lov)->obd_name, + lsm->lsm_entries[i]->lsme_pattern); +dump_lsm (D_ERROR , lsm); +return -EIO ; + } + lle->lle_extent = lsm->lsm_entries[i]->lsme_extent; + } + + i =0 ; + lov_foreach_layout_entry(lov, lle) {/** * If the component has not been init-ed on MDS side, for * PFL layout, we'd know that the components beyond this one * will be dynamically init-ed later on file write/trunc ops. */ -if (!lsm_entry_inited (lsm, i)) -continue ; - - result =lov_init_raid0 (env, dev, lov, i, &le->lle_raid0); -if (result <0 ) -break ; +if (lsm_entry_inited (lsm, i)) { + result = lle->lle_comp_ops->lco_init (env, dev, lov, i, + conf, lle); +if (result <0 ) +break ; -LASSERT (ergo (psz >0 , psz == result)); - psz = result; +LASSERT (ergo (psz >0 , psz == result)); + psz = result; + } + i++; }if (psz >0 )cl_object_header (&lov->lo_cl)->coh_page_bufsize += psz;@@ -331,10 +670,19 @@ static int lov_init_composite(const struct lu_env *env, struct lov_device *dev,return result >0 ?0 : result; } -static int lov_init_released (const struct lu_env *env,struct lov_device *dev, -struct lov_object *lov,struct lov_stripe_md *lsm, +static int lov_init_empty (const struct lu_env *env,struct lov_device *dev, +struct lov_object *lov,struct lov_stripe_md *lsm, +const struct cl_object_conf *conf, +union lov_layout_state *state) +{ +return 0 ; +} + +static int lov_init_released (const struct lu_env *env, +struct lov_device *dev,struct lov_object *lov, +struct lov_stripe_md *lsm,const struct cl_object_conf *conf, -union lov_layout_state *state) +union lov_layout_state *state) {LASSERT (lsm);LASSERT (lsm->lsm_is_released);@@ -344,41 +692,6 @@ static int lov_init_released(const struct lu_env *env, struct lov_device *dev,return 0 ; } -static struct cl_object *lov_find_subobj (const struct lu_env *env, -struct lov_object *lov, -struct lov_stripe_md *lsm, -int index) -{ -struct lov_device *dev =lu2lov_dev (lov2lu (lov)->lo_dev); -struct lov_thread_info *lti =lov_env_info (env); -struct lu_fid *ofid = <i->lti_fid; -int stripe =lov_comp_stripe (index); -int entry =lov_comp_entry (index); -struct cl_object *result =NULL ; -struct cl_device *subdev; -struct lov_oinfo *oinfo; -int ost_idx; -int rc; - -if (lov->lo_type !=LLT_COMP ) - goto out; - -if (entry >= lsm->lsm_entry_count || - stripe >= lsm->lsm_entries[entry]->lsme_stripe_count) - goto out; - - oinfo = lsm->lsm_entries[entry]->lsme_oinfo[stripe]; - ost_idx = oinfo->loi_ost_idx; - rc =ostid_to_fid (ofid, &oinfo->loi_oi, ost_idx); -if (rc) - goto out; - - subdev =lovsub2cl_dev (dev->ld_target[ost_idx]); - result =lov_sub_find (env, subdev, ofid,NULL ); -out: -return result ? result :ERR_PTR (-EINVAL ); -} -static int lov_delete_empty (const struct lu_env *env,struct lov_object *lov,union lov_layout_state *state) {@@ -388,75 +701,6 @@ static int lov_delete_empty(const struct lu_env *env, struct lov_object *lov,return 0 ; } -static void lov_subobject_kill (const struct lu_env *env,struct lov_object *lov, -struct lov_layout_raid0 *r0, -struct lovsub_object *los,int idx) -{ -struct cl_object *sub; -struct lu_site *site; -wait_queue_head_t *wq; -wait_queue_entry_t *waiter; - -LASSERT (r0->lo_sub[idx] == los); - - sub =lovsub2cl (los); - site = sub->co_lu.lo_dev->ld_site; - wq =lu_site_wq_from_fid (site, &sub->co_lu.lo_header->loh_fid); - -cl_object_kill (env, sub); -/* release a reference to the sub-object and ... */ -lu_object_ref_del (&sub->co_lu,"lov-parent" , lov); -cl_object_put (env, sub); - -/* ... wait until it is actually destroyed---sub-object clears its -* ->lo_sub[] slot in lovsub_object_fini() -*/ -if (r0->lo_sub[idx] == los) { - waiter = &lov_env_info (env)->lti_waiter; -init_waitqueue_entry (waiter, current); -add_wait_queue (wq, waiter); -set_current_state (TASK_UNINTERRUPTIBLE ); -while (1 ) { -/* this wait-queue is signaled at the end of -* lu_object_free(). -*/ -set_current_state (TASK_UNINTERRUPTIBLE ); -spin_lock (&r0->lo_sub_lock); -if (r0->lo_sub[idx] == los) { -spin_unlock (&r0->lo_sub_lock); -schedule (); - }else { -spin_unlock (&r0->lo_sub_lock); -set_current_state (TASK_RUNNING ); -break ; - } - } -remove_wait_queue (wq, waiter); - } -LASSERT (!r0->lo_sub[idx]); -} - -static void lov_delete_raid0 (const struct lu_env *env,struct lov_object *lov, -struct lov_layout_raid0 *r0) -{ -if (r0->lo_sub) { -int i; - -for (i =0 ; i < r0->lo_nr; ++i) { -struct lovsub_object *los = r0->lo_sub[i]; - -if (los) { -cl_object_prune (env, &los->lso_cl); -/* -* If top-level object is to be evicted from -* the cache, so are its sub-objects. -*/ -lov_subobject_kill (env, lov, r0, los, i); - } - } - } -} -static int lov_delete_composite (const struct lu_env *env,struct lov_object *lov,union lov_layout_state *state)@@ -469,7 +713,7 @@ static int lov_delete_composite(const struct lu_env *env,lov_layout_wait (env, lov);if (comp->lo_entries) lov_foreach_layout_entry(lov, entry) -lov_delete_raid0 (env, lov, &entry->lle_raid0); +lov_delete_raid0 (env, lov, entry);return 0 ; }@@ -480,15 +724,6 @@ static void lov_fini_empty(const struct lu_env *env, struct lov_object *lov,LASSERT (lov->lo_type ==LLT_EMPTY || lov->lo_type ==LLT_RELEASED ); } -static void lov_fini_raid0 (const struct lu_env *env, -struct lov_layout_raid0 *r0) -{ -if (r0->lo_sub) { -kvfree (r0->lo_sub); - r0->lo_sub =NULL ; - } -} -static void lov_fini_composite (const struct lu_env *env,struct lov_object *lov,union lov_layout_state *state)@@ -499,7 +734,7 @@ static void lov_fini_composite(const struct lu_env *env,struct lov_layout_entry *entry;lov_foreach_layout_entry (lov, entry) -lov_fini_raid0 (env, &entry->lle_raid0); + entry->lle_comp_ops->lco_fini(env, entry);kvfree (comp->lo_entries); comp->lo_entries =NULL ;@@ -523,24 +758,6 @@ static int lov_print_empty(const struct lu_env *env, void *cookie,return 0 ; } -static int lov_print_raid0 (const struct lu_env *env,void *cookie, -lu_printer_t p,struct lov_layout_raid0 *r0) -{ -int i; - -for (i =0 ; i < r0->lo_nr; ++i) { -struct lu_object *sub; - -if (r0->lo_sub[i]) { - sub =lovsub2lu (r0->lo_sub[i]); -lu_object_print (env, cookie, p, sub); - }else { - (*p)(env, cookie,"sub %d absent\n" , i); - } - } -return 0 ; -} -static int lov_print_composite (const struct lu_env *env,void *cookie,lu_printer_t p,const struct lu_object *o) {@@ -556,12 +773,15 @@ static int lov_print_composite(const struct lu_env *env, void *cookie,for (i =0 ; i < lsm->lsm_entry_count; i++) {struct lov_stripe_md_entry *lse = lsm->lsm_entries[i]; +struct lov_layout_entry *lle =lov_entry (lov, i); - (*p)(env, cookie,DEXT ": { 0x%08X, %u, %u, %#x, %u, %u }\n" , + (*p)(env, cookie, +DEXT ": { 0x%08X, %u, %#x, %u, %#x, %u, %u }\n" ,PEXT (&lse->lsme_extent), lse->lsme_magic, - lse->lsme_id, lse->lsme_layout_gen, lse->lsme_flags, - lse->lsme_stripe_count, lse->lsme_stripe_size); -lov_print_raid0 (env, cookie, p,lov_r0 (lov, i)); + lse->lsme_id, lse->lsme_pattern, lse->lsme_layout_gen, + lse->lsme_flags, lse->lsme_stripe_count, + lse->lsme_stripe_size); +lov_print_raid0 (env, cookie, p, lle); }return 0 ;@@ -595,52 +815,6 @@ static int lov_attr_get_empty(const struct lu_env *env, struct cl_object *obj,return 0 ; } -static int lov_attr_get_raid0 (const struct lu_env *env,struct lov_object *lov, -unsigned int index,struct lov_layout_raid0 *r0) -{ -struct lov_stripe_md *lsm = lov->lo_lsm; -struct ost_lvb *lvb = &lov_env_info (env)->lti_lvb; -struct cl_attr *attr = &r0->lo_attr; -int result =0 ; -u64 kms =0 ; - -if (r0->lo_attr_valid) -return 0 ; - -memset (lvb,0 ,sizeof (*lvb)); - -/* XXX: timestamps can be negative by sanity:test_39m, -* how can it be? -*/ - lvb->lvb_atime =LLONG_MIN ; - lvb->lvb_ctime =LLONG_MIN ; - lvb->lvb_mtime =LLONG_MIN ; - -/* -* XXX that should be replaced with a loop over sub-objects, -* doing cl_object_attr_get() on them. But for now, let's -* reuse old lov code. -*/ - -/* -* XXX take lsm spin-lock to keep lov_merge_lvb_kms() -* happy. It's not needed, because new code uses -* ->coh_attr_guard spin-lock to protect consistency of -* sub-object attributes. -*/ -lov_stripe_lock (lsm); - result =lov_merge_lvb_kms (lsm, index, lvb, &kms); -lov_stripe_unlock (lsm); -if (result) -return result; - -cl_lvb2attr (attr, lvb); - attr->cat_kms = kms; - r0->lo_attr_valid =1 ; - -return result; -} -static int lov_attr_get_composite (const struct lu_env *env,struct cl_object *obj,struct cl_attr *attr)@@ -653,19 +827,22 @@ static int lov_attr_get_composite(const struct lu_env *env, attr->cat_size =0 ; attr->cat_blocks =0 ; lov_foreach_layout_entry(lov, entry) { -struct lov_layout_raid0 *r0 = &entry->lle_raid0; -struct cl_attr *lov_attr = &r0->lo_attr; +struct cl_attr *lov_attr =NULL ;/* PFL: This component has not been init-ed. */ if (!lsm_entry_inited (lov->lo_lsm, index))break ; - result =lov_attr_get_raid0 (env, lov, index, r0); -if (result !=0 ) -break ; + result = entry->lle_comp_ops->lco_getattr (env, lov, index, + entry, &lov_attr); +if (result <0 ) +return result; index++; +if (!lov_attr) +continue ; +/* merge results */ attr->cat_blocks += lov_attr->cat_blocks;if (attr->cat_size < lov_attr->cat_size)@@ -679,7 +856,7 @@ static int lov_attr_get_composite(const struct lu_env *env,if (attr->cat_mtime < lov_attr->cat_mtime) attr->cat_mtime = lov_attr->cat_mtime; } -return result; +return 0 ; }static const struct lov_layout_operations lov_dispatch[] = {@@ -1235,6 +1412,49 @@ struct fiemap_state {bool fs_enough; }; +static struct cl_object *lov_find_subobj (const struct lu_env *env, +struct lov_object *lov, +struct lov_stripe_md *lsm, +int index) +{ +struct lov_device *dev =lu2lov_dev (lov2lu (lov)->lo_dev); +struct lov_thread_info *lti =lov_env_info (env); +struct lu_fid *ofid = <i->lti_fid; +struct lov_oinfo *oinfo; +struct cl_device *subdev; +int entry =lov_comp_entry (index); +int stripe =lov_comp_stripe (index); +int ost_idx; +int rc; +struct cl_object *result; + +if (lov->lo_type !=LLT_COMP ) { + result =NULL ; + goto out; + } + +if (entry >= lsm->lsm_entry_count || + stripe >= lsm->lsm_entries[entry]->lsme_stripe_count) { + result =NULL ; + goto out; + } + + oinfo = lsm->lsm_entries[entry]->lsme_oinfo[stripe]; + ost_idx = oinfo->loi_ost_idx; + rc =ostid_to_fid (ofid, &oinfo->loi_oi, ost_idx); +if (rc !=0 ) { + result =NULL ; + goto out; + } + + subdev =lovsub2cl_dev (dev->ld_target[ost_idx]); + result =lov_sub_find (env, subdev, ofid,NULL ); +out: +if (!result) + result =ERR_PTR (-EINVAL ); +return result; +} + static int fiemap_for_stripe(const struct lu_env *env, struct cl_object *obj, struct lov_stripe_md *lsm, struct fiemap *fiemap, size_t *buflen, struct ll_fiemap_info_key *fmkey,@@ -1457,6 +1677,12 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj, } } +/* No support for DOM layout yet. */ +if (lsme_is_dom (lsm->lsm_entries[0 ])) { + rc = -ENOTSUPP ; + goto out_lsm; + } +if (lsm->lsm_is_released) {if (fiemap->fm_start < fmkey->lfik_oa.o_size) {/** fs/lustre/lov/lov_offset.c +3
-0@@ -43,6 +43,9 @@ static u64 stripe_width(struct lov_stripe_md *lsm, unsigned int index)LASSERT (index < lsm->lsm_entry_count); +if (lsme_is_dom (entry)) +return (loff_t )entry->lsme_stripe_size; +return entry->lsme_stripe_size * entry->lsme_stripe_count; }fs/lustre/mdc/mdc_request.c +6
-1@@ -2265,7 +2265,12 @@ static int mdc_set_info_async(const struct lu_env *env,return 0 ; } -CERROR ("Unknown key %s\n" , (char *)key); +/* TODO: these OSC-related keys are ignored for now */ +if (KEY_IS (KEY_CHECKSUM ) ||KEY_IS (KEY_CACHE_SET ) || +KEY_IS (KEY_CACHE_LRU_SHRINK ) ||KEY_IS (KEY_GRANT_SHRINK )) +return 0 ; + +CERROR ("%s: Unknown key %s\n" , exp->exp_obd->obd_name, (char *)key);return -EINVAL ; }fs/lustre/obdclass/obd_config.c +35
-1@@ -972,7 +972,6 @@ int class_process_config(struct lustre_cfg *lcfg) err = -EINVAL ; goto out; } -switch (lcfg->lcfg_command) {case LCFG_SETUP : { err =class_setup (obd, lcfg);@@ -1020,6 +1019,41 @@ int class_process_config(struct lustre_cfg *lcfg) err =0 ; goto out; } +/* Process config log ADD_MDC record twice to add MDC also to LOV +* for Data-on-MDT: +* +* add 0:lustre-clilmv 1:lustre-MDT0000_UUID 2:0 3:1 +* 4:lustre-MDT0000-mdc_UUID +*/ +case LCFG_ADD_MDC : { +struct obd_device *lov_obd; +char *clilmv; + + err =obd_process_config (obd,sizeof (*lcfg), lcfg); +if (err) + goto out; + +/* make sure this is client LMV log entry */ + clilmv =strstr (lustre_cfg_string (lcfg,0 ),"clilmv" ); +if (!clilmv) + goto out; + +/* replace 'lmv' with 'lov' name to address LOV device and +* process llog record to add MDC there. +*/ + clilmv[4 ] ='o' ; + lov_obd =class_name2obd (lustre_cfg_string (lcfg,0 )); +if (!lov_obd) { + err = -ENOENT ; +CERROR ("%s: Cannot find LOV by %s name, rc = %d\n" , + obd->obd_name,lustre_cfg_string (lcfg,0 ), err); + }else { + err =obd_process_config (lov_obd,sizeof (*lcfg), lcfg); + } +/* restore 'lmv' name */ + clilmv[4 ] ='m' ; + goto out; + }default : { err =obd_process_config (obd,sizeof (*lcfg), lcfg); goto out;fs/lustre/ptlrpc/wiretest.c +2
-2@@ -1479,8 +1479,8 @@ void lustre_assert_wire_constants(void) (unsigned int)LOV_PATTERN_RAID0);LASSERTF (LOV_PATTERN_RAID1 ==0x00000002UL ,"found 0x%.8xUL\n" , (unsigned int )LOV_PATTERN_RAID1 ); -LASSERTF (LOV_PATTERN_FIRST ==0x00000100UL ,"found 0x%.8xUL\n" , - (unsigned int )LOV_PATTERN_FIRST ); +LASSERTF (LOV_PATTERN_MDT ==0x00000100UL ,"found 0x%.8xUL\n" , + (unsigned int )LOV_PATTERN_MDT );LASSERTF (LOV_PATTERN_CMOBD ==0x00000200UL ,"found 0x%.8xUL\n" , (unsigned int )LOV_PATTERN_CMOBD );include/uapi/linux/lustre/lustre_user.h +1
-1@@ -337,7 +337,7 @@ enum ll_lease_type {#define LOV_PATTERN_RAID0 0x001#define LOV_PATTERN_RAID1 0x002 -#define LOV_PATTERN_FIRST 0x100 +#define LOV_PATTERN_MDT 0x100#define LOV_PATTERN_CMOBD 0x200#define LOV_PATTERN_F_MASK 0xffff0000Signature · 1 line
1.8.3.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20191001/be422fc2/attachment-0001.sig>
PATCH 43/151 lustre: lov: add MDT target to the LOV device
Quoted 20 lines
Quoted 12 lines
From: Mikhal Pershin <[email protected]>
MDC becomes LOV target like OSC for Data-on-MDT needs. Patch does the following:
- new composite layout entry type is added -LLT_DOMto describe Data-on-MDT striping.
- LOV process config log and checks for MDC targets organizing them separately from OSCs
- LOV operations are changed where needed to understand new layout entry typeQuoted 7 lines
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit:8b352709a66f("LU-3285 lov: add MDT target to the LOV device")Signed-off-by Mikhal Pershin <[email protected]>Reviewed-on: https://review.whamcloud.com/28010
Reviewed-by Jinshan Xiong <[email protected]>Reviewed-by Andreas Dilger <[email protected]>Signed-off-by James Simmons <[email protected]>
Quoted 4 lines
Hi James,
you appear to have merged (most of) my
lustre: usewait_event()inlov_subobject_kill()
patch into this. What that intentional?
No I missed that. It was a direct port from your lustre-testing tree. It would be best to break out the change. Let me push that work to OpenSFS tree.
Quoted 1,740 lines
NeilBrown
Quoted 1,737 lines
--- fs/lustre/include/obd.h | 8 + fs/lustre/lmv/lmv_obd.c | 2 +- fs/lustre/lov/lov_cl_internal.h | 76 +++- fs/lustre/lov/lov_dev.c | 276 +++++++++++-- fs/lustre/lov/lov_ea.c | 20 +- fs/lustre/lov/lov_internal.h | 7 + fs/lustre/lov/lov_io.c | 6 +- fs/lustre/lov/lov_obd.c | 39 +- fs/lustre/lov/lov_object.c | 696 +++++++++++++++++++++----------- fs/lustre/lov/lov_offset.c | 3 + fs/lustre/mdc/mdc_request.c | 7 +- fs/lustre/obdclass/obd_config.c | 36 +- fs/lustre/ptlrpc/wiretest.c | 4 +- include/uapi/linux/lustre/lustre_user.h | 2 +- 14 files changed, 883 insertions(+), 299 deletions(-)fs/lustre/include/obd.h +8
-0@@ -381,6 +381,11 @@ struct lov_tgt_desc { ltd_reap:1 ;/* should this target be deleted */ }; +struct lov_md_tgt_desc { +struct obd_device *lmtd_mdc; +u32 lmtd_index; +}; +struct lov_obd {struct lov_desc desc;struct lov_tgt_desc **lov_tgts;/* sparse array */ @@ -403,10 +408,13 @@ struct lov_obd {struct rw_semaphore lov_notify_lock;struct kobject *lov_tgts_kobj; +/* Data-on-MDT: MDC array */ +struct lov_md_tgt_desc *lov_mdc_tgts; };struct lmv_tgt_desc {struct obd_uuid ltd_uuid; +struct obd_device *ltd_obd;struct obd_export *ltd_exp;u32 ltd_idx;struct mutex ltd_fid_mutex;fs/lustre/lmv/lmv_obd.c +1
-1@@ -389,7 +389,7 @@ static int lmv_add_target(struct obd_device *obd, struct obd_uuid *uuidp,if ((index < lmv->tgts_size) && lmv->tgts[index]) { tgt = lmv->tgts[index]; -CERROR ("%s: UUID %s already assigned at LOV target index %d: rc = %d\n" , +CERROR ("%s: UUID %s already assigned at LMV target index %d: rc = %d\n" , obd->obd_name,obd_uuid2str (&tgt->ltd_uuid), index, -EEXIST );mutex_unlock (&lmv->lmv_init_mutex);fs/lustre/lov/lov_cl_internal.h +71
-5@@ -91,6 +91,12 @@ enum lov_device_flags {* Upper half. */ +/* Data-on-MDT array item in lov_device::ld_md_tgts[] */ +struct lovdom_device { +struct cl_device *ldm_mdc; +int ldm_idx; +}; +struct lov_device {/* * XXX Locking of lov-private data is missing. @@ -101,6 +107,13 @@ struct lov_device {u32 ld_target_nr;struct lovsub_device **ld_target;u32 ld_flags; + +/* Data-on-MDT devices */ +u32 ld_md_tgts_nr; +struct lovdom_device *ld_md_tgts; +struct obd_device *ld_lmv; +/* LU site for subdevices */ +struct lu_site ld_site; };/** @@ -129,6 +142,34 @@ static inline char *llt2str(enum lov_layout_type llt)return "" ; } +/** +* Return lov_layout_entry_type associated with a given composite layout +* entry. +*/ +static inline u32 lov_entry_type (struct lov_stripe_md_entry *lsme) +{ +if ((lov_pattern (lsme->lsme_pattern) ==LOV_PATTERN_RAID0 ) || + (lov_pattern (lsme->lsme_pattern) ==LOV_PATTERN_MDT )) +return lov_pattern (lsme->lsme_pattern); +return 0 ; +} + +struct lov_layout_entry ; +struct lov_object ; +struct lov_lock_sub ; + +struct lov_comp_layout_entry_ops { +int (*lco_init)(const struct lu_env *env,struct lov_device *dev, +struct lov_object *lov,unsigned int index, +const struct cl_object_conf *conf, +struct lov_layout_entry *lle); +void (*lco_fini)(const struct lu_env *env, +struct lov_layout_entry *lle); +int (*lco_getattr)(const struct lu_env *env,struct lov_object *obj, +unsigned int index,struct lov_layout_entry *lle, +struct cl_attr **attr); +}; +struct lov_layout_raid0 {unsigned int lo_nr;/** @@ -165,6 +206,25 @@ struct lov_layout_raid0 {struct cl_attr lo_attr; }; +struct lov_layout_dom { +/* keep this always at first place so DOM layout entry +* can be addressed also as RAID0 after initialization. +*/ +struct lov_layout_raid0 lo_dom_r0; +struct lovsub_object *lo_dom; +struct lov_oinfo *lo_loi; +}; + +struct lov_layout_entry { +u32 lle_type; +struct lu_extent lle_extent; +struct lov_comp_layout_entry_ops *lle_comp_ops; +union { +struct lov_layout_raid0 lle_raid0; +struct lov_layout_dom lle_dom; + }; +}; +/** * lov-specific file state. * @@ -220,13 +280,10 @@ struct lov_object { } released;struct lov_layout_composite {/** -* Current valid entry count of lo_entries. +* Current valid entry count of entries. */ unsigned int lo_entry_count; -struct lov_layout_entry { -struct lu_extent lle_extent; -struct lov_layout_raid0 lle_raid0; - } *lo_entries; +struct lov_layout_entry *lo_entries; } composite; } u;/** @@ -633,6 +690,15 @@ static inline struct lov_thread_info *lov_env_info(const struct lu_env *env)return info; } +static inline struct lov_layout_entry *lov_entry (struct lov_object *lov,int i) +{ +LASSERT (lov->lo_type ==LLT_COMP ); +LASSERTF (i < lov->u.composite.lo_entry_count, +"entry %d entry_count %d" , i, lov->u.composite.lo_entry_count); + +return &lov->u.composite.lo_entries[i]; +} +static inline struct lov_layout_raid0 *lov_r0 (struct lov_object *lov,int i) {LASSERT (lov->lo_type ==LLT_COMP );fs/lustre/lov/lov_dev.c +239
-37@@ -146,23 +146,55 @@ struct lu_context_key lov_session_key = {/* type constructor/destructor: lov_type_{init,fini,start,stop}() */ LU_TYPE_INIT_FINI (lov, &lov_key, &lov_session_key); + +static int lov_mdc_dev_init (const struct lu_env *env,struct lov_device *ld, +struct lu_device *mdc_dev,u32 idx,u32 nr) +{ +struct cl_device *cl; + + cl =cl_type_setup (env, &ld->ld_site, &lovsub_device_type, + mdc_dev); +if (IS_ERR (cl)) +return PTR_ERR (cl); + + ld->ld_md_tgts[nr].ldm_mdc = cl; + ld->ld_md_tgts[nr].ldm_idx = idx; +return 0 ; +} +static struct lu_device *lov_device_fini (const struct lu_env *env,struct lu_device *d) { -int i;struct lov_device *ld =lu2lov_dev (d); +int i;LASSERT (ld->ld_lov); -if (!ld->ld_target) -return NULL ; - lov_foreach_target(ld, i) { -struct lovsub_device *lsd; +if (ld->ld_lmv) { +class_decref (ld->ld_lmv,"lov" , d); + ld->ld_lmv =NULL ; + } + +if (ld->ld_md_tgts) { +for (i =0 ; i < ld->ld_md_tgts_nr; i++) { +if (!ld->ld_md_tgts[i].ldm_mdc) +continue ; - lsd = ld->ld_target[i]; -if (lsd) { -cl_stack_fini (env,lovsub2cl_dev (lsd)); - ld->ld_target[i] =NULL ; +cl_stack_fini (env, ld->ld_md_tgts[i].ldm_mdc); + ld->ld_md_tgts[i].ldm_mdc =NULL ; + ld->ld_lov->lov_mdc_tgts[i].lmtd_mdc =NULL ; + } + } + +if (ld->ld_target) { + lov_foreach_target(ld, i) { +struct lovsub_device *lsd; + + lsd = ld->ld_target[i]; +if (lsd) { +cl_stack_fini (env,lovsub2cl_dev (lsd)); + ld->ld_target[i] =NULL ; + } } }return NULL ;@@ -175,9 +207,28 @@ static int lov_device_init(const struct lu_env *env, struct lu_device *d,int i;int rc =0 ; -LASSERT (d->ld_site); +/* check all added already MDC subdevices and initialize them */ +for (i =0 ; i < ld->ld_md_tgts_nr; i++) { +struct obd_device *mdc; +u32 idx; + + mdc = ld->ld_lov->lov_mdc_tgts[i].lmtd_mdc; + idx = ld->ld_lov->lov_mdc_tgts[i].lmtd_index; + +if (!mdc) +continue ; + + rc =lov_mdc_dev_init (env, ld, mdc->obd_lu_dev, idx, i); +if (rc) { +CERROR ("%s: failed to add MDC %s as target: rc = %d\n" , + d->ld_obd->obd_name, +obd_uuid2str (&mdc->obd_uuid), rc); + goto out_err; + } + } +if (!ld->ld_target) -return rc; +return 0 ; lov_foreach_target(ld, i) {struct lovsub_device *lsd;@@ -188,21 +239,21 @@ static int lov_device_init(const struct lu_env *env, struct lu_device *d,if (!desc)continue ; - cl =cl_type_setup (env, d->ld_site, &lovsub_device_type, + cl =cl_type_setup (env, &ld->ld_site, &lovsub_device_type, desc->ltd_obd->obd_lu_dev);if (IS_ERR (cl)) { rc =PTR_ERR (cl); -break ; + goto out_err; } + lsd =cl2lovsub_dev (cl); ld->ld_target[i] = lsd; } + ld->ld_flags |=LOV_DEV_INITIALIZED ; +return 0 ; -if (rc) -lov_device_fini (env, d); -else - ld->ld_flags |=LOV_DEV_INITIALIZED ; - +out_err: +lu_device_fini (d);return rc; }@@ -211,8 +262,17 @@ static struct lu_device *lov_device_free(const struct lu_env *env, {struct lov_device *ld =lu2lov_dev (d); +lu_site_fini (&ld->ld_site); +cl_device_fini (lu2cl_dev (d));kfree (ld->ld_target); + ld->ld_target =NULL ; +kfree (ld->ld_md_tgts); + ld->ld_md_tgts =NULL ; +/* free array of MDCs */ +kfree (ld->ld_lov->lov_mdc_tgts); + ld->ld_lov->lov_mdc_tgts =NULL ; +kfree (ld);return NULL ; }@@ -277,9 +337,7 @@ static int lov_cl_add_target(const struct lu_env *env, struct lu_device *dev, rc =lov_expand_targets (env, ld);if (rc ==0 && ld->ld_flags &LOV_DEV_INITIALIZED ) { -LASSERT (dev->ld_site); - - cl =cl_type_setup (env, dev->ld_site, &lovsub_device_type, + cl =cl_type_setup (env, &ld->ld_site, &lovsub_device_type, tgt->ltd_obd->obd_lu_dev);if (!IS_ERR (cl)) { lsd =cl2lovsub_dev (cl);@@ -297,6 +355,84 @@ static int lov_cl_add_target(const struct lu_env *env, struct lu_device *dev,return rc; } +/** +* Add new MDC target device in LOV. +* +* This function is part of the configuration log processing. It adds new MDC +* device to the MDC device array indexed by their indexes. +* +* @env execution environment +* @d LU device of LOV device +* @mdc MDC device to add +* @idx MDC device index +* +* Return: 0 if successful +* negative value on error +*/ +static int lov_add_mdc_target (const struct lu_env *env,struct lu_device *d, +struct obd_device *mdc,u32 idx) +{ +struct lov_device *ld =lu2lov_dev (d); +struct obd_device *lov_obd = d->ld_obd; +struct obd_device *lmv_obd; +int next; +int rc =0 ; + +LASSERT (mdc); +if (ld->ld_md_tgts_nr ==LOV_MDC_TGT_MAX ) { +/* If the maximum value of LOV_MDC_TGT_MAX will become too +* small then all MD target handling must be rewritten in LOD +* manner, check lod_add_device() and related functionality. +*/ +CERROR ("%s: cannot serve more than %d MDC devices\n" , + lov_obd->obd_name,LOV_MDC_TGT_MAX ); +return -ERANGE ; + } + +/* grab FLD from lmv, do that here, when first MDC is added +* to be sure LMV is set up and can be found +*/ +if (!ld->ld_lmv) { + next =0 ; +while ((lmv_obd =class_devices_in_group (&lov_obd->obd_uuid, + &next)) !=NULL ) { +if ((strncmp (lmv_obd->obd_type->typ_name, +LUSTRE_LMV_NAME , +strlen (LUSTRE_LMV_NAME )) ==0 )) +break ; + } +if (!lmv_obd) { +CERROR ("%s: cannot find LMV OBD by UUID (%s)\n" , + lov_obd->obd_name, +obd_uuid2str (&lmv_obd->obd_uuid)); +return -ENODEV ; + } +spin_lock (&lmv_obd->obd_dev_lock); +class_incref (lmv_obd,"lov" , ld); +spin_unlock (&lmv_obd->obd_dev_lock); + ld->ld_lmv = lmv_obd; + } + +LASSERT (!lov_obd->u.lov.lov_mdc_tgts[ld->ld_md_tgts_nr].lmtd_mdc); + +if (ld->ld_flags &LOV_DEV_INITIALIZED ) { + rc =lov_mdc_dev_init (env, ld, mdc->obd_lu_dev, idx, + ld->ld_md_tgts_nr); +if (rc) { +CERROR ("%s: failed to add MDC %s as target: rc = %d\n" , + lov_obd->obd_name,obd_uuid2str (&mdc->obd_uuid), + rc); +return rc; + } + } + + lov_obd->u.lov.lov_mdc_tgts[ld->ld_md_tgts_nr].lmtd_mdc = mdc; + lov_obd->u.lov.lov_mdc_tgts[ld->ld_md_tgts_nr].lmtd_index = idx; + ld->ld_md_tgts_nr++; + +return rc; +} +static int lov_process_config (const struct lu_env *env,struct lu_device *d,struct lustre_cfg *cfg) {@@ -309,23 +445,52 @@ static int lov_process_config(const struct lu_env *env,lov_tgts_getref (obd); cmd = cfg->lcfg_command; + rc =lov_process_config_base (d->ld_obd, cfg, &index, &gen); -if (rc ==0 ) { -switch (cmd) { -case LCFG_LOV_ADD_OBD : -case LCFG_LOV_ADD_INA : - rc =lov_cl_add_target (env, d, index); -if (rc !=0 ) -lov_del_target (d->ld_obd, index,NULL ,0 ); -break ; -case LCFG_LOV_DEL_OBD : -lov_cl_del_target (env, d, index); -break ; +if (rc <0 ) + goto out; + +switch (cmd) { +case LCFG_LOV_ADD_OBD : +case LCFG_LOV_ADD_INA : + rc =lov_cl_add_target (env, d, index); +if (rc !=0 ) +lov_del_target (d->ld_obd, index,NULL ,0 ); +break ; +case LCFG_LOV_DEL_OBD : +lov_cl_del_target (env, d, index); +break ; +case LCFG_ADD_MDC : + { +struct obd_device *mdc; +struct obd_uuid tgt_uuid; + +/* modify_mdc_tgts add 0:lustre-clilmv 1:lustre-MDT0000_UUID +* 2:0 3:1 4:lustre-MDT0000-mdc_UUID +*/ +if (LUSTRE_CFG_BUFLEN (cfg,1 ) >sizeof (tgt_uuid.uuid)) { + rc = -EINVAL ; + goto out; } - } -lov_tgts_putref (obd); +obd_str2uuid (&tgt_uuid,lustre_cfg_buf (cfg,1 )); +if (sscanf (lustre_cfg_buf (cfg,2 ),"%d" , &index) !=1 ) { + rc = -EINVAL ; + goto out; + } + mdc =class_find_client_obd (&tgt_uuid,LUSTRE_MDC_NAME , + &obd->obd_uuid); +if (!mdc) { + rc = -ENODEV ; + goto out; + } + rc =lov_add_mdc_target (env, d, mdc, index); +break ; + } + } +out: +lov_tgts_putref (obd);return rc; }@@ -355,13 +520,50 @@ static struct lu_device *lov_device_alloc(const struct lu_env *env, obd =class_name2obd (lustre_cfg_string (cfg,0 ));LASSERT (obd); rc =lov_setup (obd, cfg); -if (rc) { -lov_device_free (env, d); -return ERR_PTR (rc); +if (rc) + goto out; + +/* Alloc MDC devices array */ +/* XXX: need dynamic allocation at some moment */ + ld->ld_md_tgts =kcalloc (LOV_MDC_TGT_MAX ,sizeof (*ld->ld_md_tgts), +GFP_NOFS ); +if (!ld->ld_md_tgts) { + rc = -ENOMEM ; + goto out; } + ld->ld_md_tgts_nr =0 ; ld->ld_lov = &obd->u.lov; + ld->ld_lov->lov_mdc_tgts = +kcalloc (LOV_MDC_TGT_MAX , +sizeof (*ld->ld_lov->lov_mdc_tgts), +GFP_NOFS ); +if (!ld->ld_lov->lov_mdc_tgts) { + rc = -ENOMEM ; + goto out_md_tgts; + } + + rc =lu_site_init (&ld->ld_site, d); +if (rc !=0 ) + goto out_mdc_tgts; + + rc =lu_site_init_finish (&ld->ld_site); +if (rc !=0 ) + goto out_site; +return d; +out_site: +lu_site_fini (&ld->ld_site); +out_mdc_tgts: +kfree (ld->ld_lov->lov_mdc_tgts); + ld->ld_lov->lov_mdc_tgts =NULL ; +out_md_tgts: +kfree (ld->ld_md_tgts); + ld->ld_md_tgts =NULL ; +out: +kfree (ld); + +return ERR_PTR (rc); }static const struct lu_device_type_operations lov_device_type_ops = {fs/lustre/lov/lov_ea.c +15
-5@@ -95,7 +95,8 @@ static int lsm_lmm_verify_v1v3(struct lov_mds_md *lmm, size_t lmm_size,return -EINVAL ; } -if (lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_RAID0 ) { +if (lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_MDT && +lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_RAID0 ) {CERROR ("bad striping pattern\n" );lov_dump_lmm_common (D_WARNING , lmm);return -EINVAL ;@@ -206,6 +207,12 @@ void lsm_free(struct lov_stripe_md *lsm) } } +/* with Data-on-MDT set maxbytes to stripe size */ +if (lsme_is_dom (lsme)) { + lov_bytes = lsme->lsme_stripe_size; + goto out_dom; + } +for (i =0 ; i < stripe_count; i++) {struct lov_tgt_desc *ltd;struct lov_oinfo *loi;@@ -253,6 +260,7 @@ void lsm_free(struct lov_stripe_md *lsm) lov_bytes = min_stripe_maxbytes * stripe_count; +out_dom:if (maxbytes) {if (lov_bytes < min_stripe_maxbytes)/* handle overflow */ *maxbytes =MAX_LFS_FILESIZE ;@@ -385,7 +393,8 @@ static int lsm_verify_comp_md_v1(struct lov_comp_md_v1 *lcm,unsigned int magic; stripe_count =le16_to_cpu (lmm->lmm_stripe_count); -if (stripe_count ==0 ) +if (stripe_count ==0 && +lov_pattern (le32_to_cpu (lmm->lmm_pattern)) !=LOV_PATTERN_MDT )return ERR_PTR (-EINVAL );/* un-instantiated lmm contains no ost id info, i.e. lov_ost_data_v1 */ @@ -474,9 +483,10 @@ static int lsm_verify_comp_md_v1(struct lov_comp_md_v1 *lcm,/* the last component hasn't been defined, or * lsm_maxbytes overflowed. */ -if (lsme->lsme_extent.e_end !=LUSTRE_EOF || - lsm->lsm_maxbytes < - (loff_t )lsme->lsme_extent.e_start) +if (!lsme_is_dom (lsme) && + (lsme->lsme_extent.e_end !=LUSTRE_EOF || + lsm->lsm_maxbytes < + (loff_t )lsme->lsme_extent.e_start)) lsm->lsm_maxbytes =MAX_LFS_FILESIZE ; } }fs/lustre/lov/lov_internal.h +7
-0@@ -57,6 +57,11 @@ struct lov_stripe_md_entry {struct lov_oinfo *lsme_oinfo[]; }; +static inline bool lsme_is_dom (struct lov_stripe_md_entry *lsme) +{ +return (lov_pattern (lsme->lsme_pattern) ==LOV_PATTERN_MDT ); +} +static inline void copy_lsm_entry (struct lov_stripe_md_entry *dst,struct lov_stripe_md_entry *src) {@@ -300,6 +305,8 @@ struct lov_stripe_md *lov_unpackmd(struct lov_obd *lov, void *buf,/* lov_cl.c */ extern struct lu_device_type lov_device_type; +#define LOV_MDC_TGT_MAX 256 +/* ost_pool methods */ int lov_ost_pool_init (struct ost_pool *op,unsigned int count);int lov_ost_pool_extend (struct ost_pool *op,unsigned int min_count);fs/lustre/lov/lov_io.c +5
-1@@ -533,7 +533,11 @@ static int lov_io_setattr_iter_init(const struct lu_env *env,if (cl_io_is_trunc (io) && lio->lis_pos >0 ) { index =lov_lsm_entry (lsm, lio->lis_pos -1 ); -if (index >0 && !lsm_entry_inited (lsm, index)) { +/* no entry found for such offset */ +if (index <0 ) { + io->ci_result = -ENODATA ; +return io->ci_result; + }else if (!lsm_entry_inited (lsm, index)) { io->ci_need_write_intent =1 ; io->ci_result = -ENODATA ;return io->ci_result;fs/lustre/lov/lov_obd.c +29
-10@@ -852,6 +852,9 @@ int lov_process_config_base(struct obd_device *obd, struct lustre_cfg *lcfg,int rc =0 ;switch (cmd = lcfg->lcfg_command) { +case LCFG_ADD_MDC : +case LCFG_DEL_MDC : +break ;case LCFG_LOV_ADD_OBD :case LCFG_LOV_ADD_INA :case LCFG_LOV_DEL_OBD : {@@ -1179,31 +1182,32 @@ static int lov_set_info_async(const struct lu_env *env, struct obd_export *exp, {struct obd_device *obddev =class_exp2obd (exp);struct lov_obd *lov = &obddev->u.lov; -u32 count; -int i, rc =0 , err;struct lov_tgt_desc *tgt; -int do_inactive =0 , no_set =0 ; +bool do_inactive = false; +bool no_set = false; +int rc =0 ; +int err; +u32 i;if (!set) { - no_set =1 ; + no_set = true; set =ptlrpc_prep_set ();if (!set)return -ENOMEM ; }lov_tgts_getref (obddev); - count = lov->desc.ld_tgt_count;if (KEY_IS (KEY_CHECKSUM )) { - do_inactive =1 ; + do_inactive = true; }else if (KEY_IS (KEY_CACHE_SET )) {LASSERT (!lov->lov_cache); lov->lov_cache = val; - do_inactive =1 ; + do_inactive = true;cl_cache_incref (lov->lov_cache); } -for (i =0 ; i < count; i++) { +for (i =0 ; i < lov->desc.ld_tgt_count; i++) { tgt = lov->lov_tgts[i];/* OST was disconnected */ @@ -1216,14 +1220,29 @@ static int lov_set_info_async(const struct lu_env *env, struct obd_export *exp, err =obd_set_info_async (env, tgt->ltd_exp, keylen, key, vallen, val, set); -if (!rc) + +if (rc ==0 ) + rc = err; + } + +/* cycle through MDC target for Data-on-MDT */ +for (i =0 ; i <LOV_MDC_TGT_MAX ; i++) { +struct obd_device *mdc; + + mdc = lov->lov_mdc_tgts[i].lmtd_mdc; +if (!mdc) +continue ; + + err =obd_set_info_async (env, mdc->obd_self_export, + keylen, key, vallen, val, set); +if (rc ==0 ) rc = err; }lov_tgts_putref (obddev);if (no_set) { err =ptlrpc_set_wait (set); -if (!rc) +if (rc ==0 ) rc = err;ptlrpc_set_destroy (set); }fs/lustre/lov/lov_object.c +461
-235@@ -90,13 +90,6 @@ static void lov_lsm_put(struct lov_stripe_md *lsm)* Lov object layout operations. * */ -static int lov_init_empty (const struct lu_env *env,struct lov_device *dev, -struct lov_object *lov,struct lov_stripe_md *lsm, -const struct cl_object_conf *conf, -union lov_layout_state *state) -{ -return 0 ; -} static struct cl_object *lov_sub_find(const struct lu_env *env, struct cl_device *dev,@@ -110,9 +103,25 @@ static struct cl_object *lov_sub_find(const struct lu_env *env,return lu2cl (o); } +static int lov_page_slice_fixup (struct lov_object *lov, +struct cl_object *stripe) +{ +struct cl_object_header *hdr =cl_object_header (&lov->lo_cl); +struct cl_object *o; + +if (!stripe) +return hdr->coh_page_bufsize - lov->lo_cl.co_slice_off - +cfs_size_round (sizeof (struct lov_page )); + +cl_object_for_each (o, stripe) + o->co_slice_off += hdr->coh_page_bufsize; + +return cl_object_header (stripe)->coh_page_bufsize; +} +static int lov_init_sub (const struct lu_env *env,struct lov_object *lov, -struct cl_object *subobj,struct lov_layout_raid0 *r0, -struct lov_oinfo *oinfo,int idx) +struct cl_object *subobj,struct lov_oinfo *oinfo, +int idx) {int stripe =lov_comp_stripe (idx);int entry =lov_comp_entry (idx);@@ -146,13 +155,14 @@ static int lov_init_sub(const struct lu_env *env, struct lov_object *lov,spin_lock (&subhdr->coh_attr_guard); parent = subhdr->coh_parent;if (!parent) { +struct lovsub_object *lso =cl2lovsub (subobj); + subhdr->coh_parent = hdr;spin_unlock (&subhdr->coh_attr_guard); subhdr->coh_nesting = hdr->coh_nesting +1 ;lu_object_ref_add (&subobj->co_lu,"lov-parent" , lov); - r0->lo_sub[stripe] =cl2lovsub (subobj); - r0->lo_sub[stripe]->lso_super = lov; - r0->lo_sub[stripe]->lso_index = idx; + lso->lso_super = lov; + lso->lso_index = idx; result =0 ; }else {struct lu_object *old_obj;@@ -183,33 +193,19 @@ static int lov_init_sub(const struct lu_env *env, struct lov_object *lov,return result; } -static int lov_page_slice_fixup (struct lov_object *lov, -struct cl_object *stripe) -{ -struct cl_object_header *hdr =cl_object_header (&lov->lo_cl); -struct cl_object *o; - -if (!stripe) -return hdr->coh_page_bufsize - lov->lo_cl.co_slice_off - -cfs_size_round (sizeof (struct lov_page )); - -cl_object_for_each (o, stripe) - o->co_slice_off += hdr->coh_page_bufsize; - -return cl_object_header (stripe)->coh_page_bufsize; -} -static int lov_init_raid0 (const struct lu_env *env,struct lov_device *dev, -struct lov_object *lov,int index, -struct lov_layout_raid0 *r0) +struct lov_object *lov,unsigned int index, +const struct cl_object_conf *conf, +struct lov_layout_entry *lle) {struct lov_stripe_md_entry *lse =lov_lse (lov, index); +struct lov_layout_raid0 *r0 = &lle->lle_raid0;struct lov_thread_info *lti =lov_env_info (env);struct cl_object_conf *subconf = <i->lti_stripe_conf;struct lu_fid *ofid = <i->lti_fid;struct cl_object *stripe;int result; -int psz; +int psz, sz;int i;spin_lock_init (&r0->lo_sub_lock);@@ -261,7 +257,7 @@ static int lov_init_raid0(const struct lu_env *env, struct lov_device *dev, goto out; } - result =lov_init_sub (env, lov, stripe, r0, oinfo, + result =lov_init_sub (env, lov, stripe, oinfo,lov_comp_index (index, i));if (result == -EAGAIN ) {/* try again */ --i;@@ -270,8 +266,9 @@ static int lov_init_raid0(const struct lu_env *env, struct lov_device *dev, }if (result ==0 ) { -int sz =lov_page_slice_fixup (lov, stripe); + r0->lo_sub[i] =cl2lovsub (stripe); + sz =lov_page_slice_fixup (lov, stripe);LASSERT (ergo (psz >0 , psz == sz)); psz = sz; }@@ -282,12 +279,333 @@ static int lov_init_raid0(const struct lu_env *env, struct lov_device *dev,return result; } +static void lov_subobject_kill (const struct lu_env *env,struct lov_object *lov, +struct lov_layout_raid0 *r0, +struct lovsub_object *los,int idx) +{ +struct cl_object *sub; +struct lu_site *site; +wait_queue_head_t *wq; + +LASSERT (r0->lo_sub[idx] == los); + + sub =lovsub2cl (los); + site = sub->co_lu.lo_dev->ld_site; + wq =lu_site_wq_from_fid (site, &sub->co_lu.lo_header->loh_fid); + +cl_object_kill (env, sub); +/* release a reference to the sub-object and ... */ +lu_object_ref_del (&sub->co_lu,"lov-parent" , lov); +cl_object_put (env, sub); + +/* ... wait until it is actually destroyed---sub-object clears its +* ->lo_sub[] slot in lovsub_object_free() +*/ +wait_event (*wq, r0->lo_sub[idx] != los); +LASSERT (!r0->lo_sub[idx]); +} + +static void lov_delete_raid0 (const struct lu_env *env,struct lov_object *lov, +struct lov_layout_entry *lle) +{ +struct lov_layout_raid0 *r0 = &lle->lle_raid0; + +if (r0->lo_sub) { +int i; + +for (i =0 ; i < r0->lo_nr; ++i) { +struct lovsub_object *los = r0->lo_sub[i]; + +if (los) { +cl_object_prune (env, &los->lso_cl); +/* +* If top-level object is to be evicted from +* the cache, so are its sub-objects. +*/ +lov_subobject_kill (env, lov, r0, los, i); + } + } + } +} + +static void lov_fini_raid0 (const struct lu_env *env, +struct lov_layout_entry *lle) +{ +struct lov_layout_raid0 *r0 = &lle->lle_raid0; + +if (r0->lo_sub) { +kvfree (r0->lo_sub); + r0->lo_sub =NULL ; + } +} + +static int lov_print_raid0 (const struct lu_env *env,void *cookie, +lu_printer_t p,const struct lov_layout_entry *lle) +{ +const struct lov_layout_raid0 *r0 = &lle->lle_raid0; +int i; + +for (i =0 ; i < r0->lo_nr; ++i) { +struct lu_object *sub; + +if (r0->lo_sub[i]) { + sub =lovsub2lu (r0->lo_sub[i]); +lu_object_print (env, cookie, p, sub); + }else { + (*p)(env, cookie,"sub %d absent\n" , i); + } + } +return 0 ; +} + +static int lov_attr_get_raid0 (const struct lu_env *env,struct lov_object *lov, +unsigned int index,struct lov_layout_entry *lle, +struct cl_attr **lov_attr) +{ +struct lov_layout_raid0 *r0 = &lle->lle_raid0; +struct lov_stripe_md *lsm = lov->lo_lsm; +struct ost_lvb *lvb = &lov_env_info (env)->lti_lvb; +struct cl_attr *attr = &r0->lo_attr; +u64 kms =0 ; +int result =0 ; + +if (r0->lo_attr_valid) { + *lov_attr = attr; +return 0 ; + } + +memset (lvb,0 ,sizeof (*lvb)); + +/* XXX: timestamps can be negative by sanity:test_39m, +* how can it be? +*/ + lvb->lvb_atime =LLONG_MIN ; + lvb->lvb_ctime =LLONG_MIN ; + lvb->lvb_mtime =LLONG_MIN ; + +/* +* XXX that should be replaced with a loop over sub-objects, +* doing cl_object_attr_get() on them. But for now, let's +* reuse old lov code. +*/ + +/* +* XXX take lsm spin-lock to keep lov_merge_lvb_kms() +* happy. It's not needed, because new code uses +* ->coh_attr_guard spin-lock to protect consistency of +* sub-object attributes. +*/ +lov_stripe_lock (lsm); + result =lov_merge_lvb_kms (lsm, index, lvb, &kms); +lov_stripe_unlock (lsm); +if (result ==0 ) { +cl_lvb2attr (attr, lvb); + attr->cat_kms = kms; + r0->lo_attr_valid =1 ; + *lov_attr = attr; + } + +return result; +} + +static struct lov_comp_layout_entry_ops raid0_ops = { + .lco_init = lov_init_raid0, + .lco_fini = lov_fini_raid0, + .lco_getattr = lov_attr_get_raid0, +}; + +static int lov_attr_get_dom (const struct lu_env *env,struct lov_object *lov, +unsigned int index,struct lov_layout_entry *lle, +struct cl_attr **lov_attr) +{ +struct lov_layout_dom *dom = &lle->lle_dom; +struct lov_oinfo *loi = dom->lo_loi; +struct cl_attr *attr = &dom->lo_dom_r0.lo_attr; + +if (dom->lo_dom_r0.lo_attr_valid) { + *lov_attr = attr; +return 0 ; + } + +if (OST_LVB_IS_ERR (loi->loi_lvb.lvb_blocks)) +return OST_LVB_GET_ERR (loi->loi_lvb.lvb_blocks); + +cl_lvb2attr (attr, &loi->loi_lvb); + attr->cat_kms = attr->cat_size > loi->loi_kms ? attr->cat_size : + loi->loi_kms; + dom->lo_dom_r0.lo_attr_valid =1 ; + *lov_attr = attr; + +return 0 ; +} + +/** +* Lookup FLD to get MDS index of the given DOM object FID. +* +* @ld LOV device +* @fid FID to lookup +* @nr index in MDC array to return back +* +* Return: 0 and @mds filled with MDS index if successful +* negative value on error +*/ +static int lov_fld_lookup (struct lov_device *ld,const struct lu_fid *fid, +u32 *nr) +{ +u32 mds_idx; +int i, rc; + + rc =fld_client_lookup (&ld->ld_lmv->u.lmv.lmv_fld,fid_seq (fid), + &mds_idx,LU_SEQ_RANGE_MDT ,NULL ); +if (rc) { +CERROR ("%s: error while looking for mds number. Seq %#llx, err = %d\n" , +lu_dev_name (cl2lu_dev (&ld->ld_cl)),fid_seq (fid), rc); +return rc; + } + +CDEBUG (D_INODE ,"FLD lookup got mds #%x for fid=" DFID "\n" , + mds_idx,PFID (fid)); + +/* find proper MDC device in the array */ +for (i =0 ; i < ld->ld_md_tgts_nr; i++) { +if (ld->ld_md_tgts[i].ldm_mdc && + ld->ld_md_tgts[i].ldm_idx == mds_idx) +break ; + } + +if (i == ld->ld_md_tgts_nr) { +CERROR ("%s: cannot find corresponding MDC device for mds #%x for fid=" DFID "\n" , +lu_dev_name (cl2lu_dev (&ld->ld_cl)), mds_idx,PFID (fid)); + rc = -EINVAL ; + }else { + *nr = i; + } +return rc; +} + +/** +* Implementation of lov_comp_layout_entry_ops::lco_init for DOM object. +* +* Init the DOM object for the first time. It prepares also RAID0 entry +* for it to use in common methods with ordinary RAID0 layout entries. +* +* @env execution environment +* @dev LOV device +* @lov LOV object +* @index Composite layout entry index in LSM +* @lle Composite LOV layout entry +*/ +static int lov_init_dom (const struct lu_env *env,struct lov_device *dev, +struct lov_object *lov,unsigned int index, +const struct cl_object_conf *conf, +struct lov_layout_entry *lle) +{ +struct lov_thread_info *lti =lov_env_info (env); +struct lov_stripe_md_entry *lsme =lov_lse (lov, index); +struct cl_object *clo; +struct lu_object *o =lov2lu (lov); +const struct lu_fid *fid =lu_object_fid (o); +struct cl_device *mdcdev; +struct lov_oinfo *loi =NULL ; +struct cl_object_conf *sconf = <i->lti_stripe_conf; +struct inode *inode = conf->coc_inode; +u32 idx =0 ; +int rc; + +LASSERT (index ==0 ); + +/* find proper MDS device */ + rc =lov_fld_lookup (dev, fid, &idx); +if (rc) +return rc; + +LASSERTF (dev->ld_md_tgts[idx].ldm_mdc, +"LOV md target[%u] is NULL\n" , idx); + +/* check lsm is DOM, more checks are needed */ +LASSERT (lsme->lsme_stripe_count ==0 ); + +/* +* Create lower cl_objects. +*/ + mdcdev = dev->ld_md_tgts[idx].ldm_mdc; + +LASSERTF (mdcdev,"non-initialized mdc subdev\n" ); + +/* DoM object has no oinfo in LSM entry, create it exclusively */ + loi =kmem_cache_zalloc (lov_oinfo_slab,GFP_NOFS ); +if (!loi) +return -ENOMEM ; + +fid_to_ostid (lu_object_fid (lov2lu (lov)), &loi->loi_oi); +/* Initialize lvb structure */ + loi->loi_lvb.lvb_mtime = inode->i_mtime.tv_sec; + loi->loi_lvb.lvb_atime = inode->i_atime.tv_sec; + loi->loi_lvb.lvb_ctime = inode->i_ctime.tv_sec; + loi->loi_lvb.lvb_blocks = inode->i_blocks; + loi->loi_lvb.lvb_size =i_size_read (inode); +if (loi->loi_lvb.lvb_size > lsme->lsme_stripe_size) + loi->loi_lvb.lvb_size = lsme->lsme_stripe_size; +loi_kms_set (loi, loi->loi_lvb.lvb_size); + + sconf->u.coc_oinfo = loi; +again: + clo =lov_sub_find (env, mdcdev, fid, sconf); +if (IS_ERR (clo)) { + rc =PTR_ERR (clo); + goto out; + } + + rc =lov_init_sub (env, lov, clo, loi,lov_comp_index (index,0 )); +if (rc == -EAGAIN )/* try again */ + goto again; +else if (rc !=0 ) + goto out; + + lle->lle_dom.lo_dom =cl2lovsub (clo); +spin_lock_init (&lle->lle_dom.lo_dom_r0.lo_sub_lock); + lle->lle_dom.lo_dom_r0.lo_nr =1 ; + lle->lle_dom.lo_dom_r0.lo_sub = &lle->lle_dom.lo_dom; + lle->lle_dom.lo_loi = loi; + + rc =lov_page_slice_fixup (lov, clo); +return rc; + +out: +kmem_cache_free (lov_oinfo_slab, loi); +return rc; +} + +/** +* Implementation of lov_layout_operations::llo_fini for DOM object. +* +* Finish the DOM object and free related memory. +* +* @env execution environment +* @lov LOV object +* @state LOV layout state +*/ +static void lov_fini_dom (const struct lu_env *env, +struct lov_layout_entry *lle) +{ +if (lle->lle_dom.lo_dom) + lle->lle_dom.lo_dom =NULL ; +kmem_cache_free (lov_oinfo_slab, lle->lle_dom.lo_loi); +} + +static struct lov_comp_layout_entry_ops dom_ops = { + .lco_init = lov_init_dom, + .lco_fini = lov_fini_dom, + .lco_getattr = lov_attr_get_dom, +}; +static int lov_init_composite (const struct lu_env *env,struct lov_device *dev,struct lov_object *lov,struct lov_stripe_md *lsm,const struct cl_object_conf *conf,union lov_layout_state *state) {struct lov_layout_composite *comp = &state->composite; +struct lov_layout_entry *lle;unsigned int entry_count;unsigned int psz =0 ;int result =0 ;@@ -306,24 +624,45 @@ static int lov_init_composite(const struct lu_env *env, struct lov_device *dev,if (!comp->lo_entries)return -ENOMEM ; +/* Initiate all entry types and extents data at first */ for (i =0 ; i < entry_count; i++) { -struct lov_layout_entry *le = &comp->lo_entries[i]; + lle = &comp->lo_entries[i]; - le->lle_extent = lsm->lsm_entries[i]->lsme_extent; + lle->lle_type =lov_entry_type (lsm->lsm_entries[i]); +switch (lle->lle_type) { +case LOV_PATTERN_RAID0 : + lle->lle_comp_ops = &raid0_ops; +break ; +case LOV_PATTERN_MDT : + lle->lle_comp_ops = &dom_ops; +break ; +default : +CERROR ("%s: unknown composite layout entry type %i\n" , +lov2obd (dev->ld_lov)->obd_name, + lsm->lsm_entries[i]->lsme_pattern); +dump_lsm (D_ERROR , lsm); +return -EIO ; + } + lle->lle_extent = lsm->lsm_entries[i]->lsme_extent; + } + + i =0 ; + lov_foreach_layout_entry(lov, lle) {/** * If the component has not been init-ed on MDS side, for * PFL layout, we'd know that the components beyond this one * will be dynamically init-ed later on file write/trunc ops. */ -if (!lsm_entry_inited (lsm, i)) -continue ; - - result =lov_init_raid0 (env, dev, lov, i, &le->lle_raid0); -if (result <0 ) -break ; +if (lsm_entry_inited (lsm, i)) { + result = lle->lle_comp_ops->lco_init (env, dev, lov, i, + conf, lle); +if (result <0 ) +break ; -LASSERT (ergo (psz >0 , psz == result)); - psz = result; +LASSERT (ergo (psz >0 , psz == result)); + psz = result; + } + i++; }if (psz >0 )cl_object_header (&lov->lo_cl)->coh_page_bufsize += psz;@@ -331,10 +670,19 @@ static int lov_init_composite(const struct lu_env *env, struct lov_device *dev,return result >0 ?0 : result; } -static int lov_init_released (const struct lu_env *env,struct lov_device *dev, -struct lov_object *lov,struct lov_stripe_md *lsm, +static int lov_init_empty (const struct lu_env *env,struct lov_device *dev, +struct lov_object *lov,struct lov_stripe_md *lsm, +const struct cl_object_conf *conf, +union lov_layout_state *state) +{ +return 0 ; +} + +static int lov_init_released (const struct lu_env *env, +struct lov_device *dev,struct lov_object *lov, +struct lov_stripe_md *lsm,const struct cl_object_conf *conf, -union lov_layout_state *state) +union lov_layout_state *state) {LASSERT (lsm);LASSERT (lsm->lsm_is_released);@@ -344,41 +692,6 @@ static int lov_init_released(const struct lu_env *env, struct lov_device *dev,return 0 ; } -static struct cl_object *lov_find_subobj (const struct lu_env *env, -struct lov_object *lov, -struct lov_stripe_md *lsm, -int index) -{ -struct lov_device *dev =lu2lov_dev (lov2lu (lov)->lo_dev); -struct lov_thread_info *lti =lov_env_info (env); -struct lu_fid *ofid = <i->lti_fid; -int stripe =lov_comp_stripe (index); -int entry =lov_comp_entry (index); -struct cl_object *result =NULL ; -struct cl_device *subdev; -struct lov_oinfo *oinfo; -int ost_idx; -int rc; - -if (lov->lo_type !=LLT_COMP ) - goto out; - -if (entry >= lsm->lsm_entry_count || - stripe >= lsm->lsm_entries[entry]->lsme_stripe_count) - goto out; - - oinfo = lsm->lsm_entries[entry]->lsme_oinfo[stripe]; - ost_idx = oinfo->loi_ost_idx; - rc =ostid_to_fid (ofid, &oinfo->loi_oi, ost_idx); -if (rc) - goto out; - - subdev =lovsub2cl_dev (dev->ld_target[ost_idx]); - result =lov_sub_find (env, subdev, ofid,NULL ); -out: -return result ? result :ERR_PTR (-EINVAL ); -} -static int lov_delete_empty (const struct lu_env *env,struct lov_object *lov,union lov_layout_state *state) {@@ -388,75 +701,6 @@ static int lov_delete_empty(const struct lu_env *env, struct lov_object *lov,return 0 ; } -static void lov_subobject_kill (const struct lu_env *env,struct lov_object *lov, -struct lov_layout_raid0 *r0, -struct lovsub_object *los,int idx) -{ -struct cl_object *sub; -struct lu_site *site; -wait_queue_head_t *wq; -wait_queue_entry_t *waiter; - -LASSERT (r0->lo_sub[idx] == los); - - sub =lovsub2cl (los); - site = sub->co_lu.lo_dev->ld_site; - wq =lu_site_wq_from_fid (site, &sub->co_lu.lo_header->loh_fid); - -cl_object_kill (env, sub); -/* release a reference to the sub-object and ... */ -lu_object_ref_del (&sub->co_lu,"lov-parent" , lov); -cl_object_put (env, sub); - -/* ... wait until it is actually destroyed---sub-object clears its -* ->lo_sub[] slot in lovsub_object_fini() -*/ -if (r0->lo_sub[idx] == los) { - waiter = &lov_env_info (env)->lti_waiter; -init_waitqueue_entry (waiter, current); -add_wait_queue (wq, waiter); -set_current_state (TASK_UNINTERRUPTIBLE ); -while (1 ) { -/* this wait-queue is signaled at the end of -* lu_object_free(). -*/ -set_current_state (TASK_UNINTERRUPTIBLE ); -spin_lock (&r0->lo_sub_lock); -if (r0->lo_sub[idx] == los) { -spin_unlock (&r0->lo_sub_lock); -schedule (); - }else { -spin_unlock (&r0->lo_sub_lock); -set_current_state (TASK_RUNNING ); -break ; - } - } -remove_wait_queue (wq, waiter); - } -LASSERT (!r0->lo_sub[idx]); -} - -static void lov_delete_raid0 (const struct lu_env *env,struct lov_object *lov, -struct lov_layout_raid0 *r0) -{ -if (r0->lo_sub) { -int i; - -for (i =0 ; i < r0->lo_nr; ++i) { -struct lovsub_object *los = r0->lo_sub[i]; - -if (los) { -cl_object_prune (env, &los->lso_cl); -/* -* If top-level object is to be evicted from -* the cache, so are its sub-objects. -*/ -lov_subobject_kill (env, lov, r0, los, i); - } - } - } -} -static int lov_delete_composite (const struct lu_env *env,struct lov_object *lov,union lov_layout_state *state)@@ -469,7 +713,7 @@ static int lov_delete_composite(const struct lu_env *env,lov_layout_wait (env, lov);if (comp->lo_entries) lov_foreach_layout_entry(lov, entry) -lov_delete_raid0 (env, lov, &entry->lle_raid0); +lov_delete_raid0 (env, lov, entry);return 0 ; }@@ -480,15 +724,6 @@ static void lov_fini_empty(const struct lu_env *env, struct lov_object *lov,LASSERT (lov->lo_type ==LLT_EMPTY || lov->lo_type ==LLT_RELEASED ); } -static void lov_fini_raid0 (const struct lu_env *env, -struct lov_layout_raid0 *r0) -{ -if (r0->lo_sub) { -kvfree (r0->lo_sub); - r0->lo_sub =NULL ; - } -} -static void lov_fini_composite (const struct lu_env *env,struct lov_object *lov,union lov_layout_state *state)@@ -499,7 +734,7 @@ static void lov_fini_composite(const struct lu_env *env,struct lov_layout_entry *entry;lov_foreach_layout_entry (lov, entry) -lov_fini_raid0 (env, &entry->lle_raid0); + entry->lle_comp_ops->lco_fini(env, entry);kvfree (comp->lo_entries); comp->lo_entries =NULL ;@@ -523,24 +758,6 @@ static int lov_print_empty(const struct lu_env *env, void *cookie,return 0 ; } -static int lov_print_raid0 (const struct lu_env *env,void *cookie, -lu_printer_t p,struct lov_layout_raid0 *r0) -{ -int i; - -for (i =0 ; i < r0->lo_nr; ++i) { -struct lu_object *sub; - -if (r0->lo_sub[i]) { - sub =lovsub2lu (r0->lo_sub[i]); -lu_object_print (env, cookie, p, sub); - }else { - (*p)(env, cookie,"sub %d absent\n" , i); - } - } -return 0 ; -} -static int lov_print_composite (const struct lu_env *env,void *cookie,lu_printer_t p,const struct lu_object *o) {@@ -556,12 +773,15 @@ static int lov_print_composite(const struct lu_env *env, void *cookie,for (i =0 ; i < lsm->lsm_entry_count; i++) {struct lov_stripe_md_entry *lse = lsm->lsm_entries[i]; +struct lov_layout_entry *lle =lov_entry (lov, i); - (*p)(env, cookie,DEXT ": { 0x%08X, %u, %u, %#x, %u, %u }\n" , + (*p)(env, cookie, +DEXT ": { 0x%08X, %u, %#x, %u, %#x, %u, %u }\n" ,PEXT (&lse->lsme_extent), lse->lsme_magic, - lse->lsme_id, lse->lsme_layout_gen, lse->lsme_flags, - lse->lsme_stripe_count, lse->lsme_stripe_size); -lov_print_raid0 (env, cookie, p,lov_r0 (lov, i)); + lse->lsme_id, lse->lsme_pattern, lse->lsme_layout_gen, + lse->lsme_flags, lse->lsme_stripe_count, + lse->lsme_stripe_size); +lov_print_raid0 (env, cookie, p, lle); }return 0 ;@@ -595,52 +815,6 @@ static int lov_attr_get_empty(const struct lu_env *env, struct cl_object *obj,return 0 ; } -static int lov_attr_get_raid0 (const struct lu_env *env,struct lov_object *lov, -unsigned int index,struct lov_layout_raid0 *r0) -{ -struct lov_stripe_md *lsm = lov->lo_lsm; -struct ost_lvb *lvb = &lov_env_info (env)->lti_lvb; -struct cl_attr *attr = &r0->lo_attr; -int result =0 ; -u64 kms =0 ; - -if (r0->lo_attr_valid) -return 0 ; - -memset (lvb,0 ,sizeof (*lvb)); - -/* XXX: timestamps can be negative by sanity:test_39m, -* how can it be? -*/ - lvb->lvb_atime =LLONG_MIN ; - lvb->lvb_ctime =LLONG_MIN ; - lvb->lvb_mtime =LLONG_MIN ; - -/* -* XXX that should be replaced with a loop over sub-objects, -* doing cl_object_attr_get() on them. But for now, let's -* reuse old lov code. -*/ - -/* -* XXX take lsm spin-lock to keep lov_merge_lvb_kms() -* happy. It's not needed, because new code uses -* ->coh_attr_guard spin-lock to protect consistency of -* sub-object attributes. -*/ -lov_stripe_lock (lsm); - result =lov_merge_lvb_kms (lsm, index, lvb, &kms); -lov_stripe_unlock (lsm); -if (result) -return result; - -cl_lvb2attr (attr, lvb); - attr->cat_kms = kms; - r0->lo_attr_valid =1 ; - -return result; -} -static int lov_attr_get_composite (const struct lu_env *env,struct cl_object *obj,struct cl_attr *attr)@@ -653,19 +827,22 @@ static int lov_attr_get_composite(const struct lu_env *env, attr->cat_size =0 ; attr->cat_blocks =0 ; lov_foreach_layout_entry(lov, entry) { -struct lov_layout_raid0 *r0 = &entry->lle_raid0; -struct cl_attr *lov_attr = &r0->lo_attr; +struct cl_attr *lov_attr =NULL ;/* PFL: This component has not been init-ed. */ if (!lsm_entry_inited (lov->lo_lsm, index))break ; - result =lov_attr_get_raid0 (env, lov, index, r0); -if (result !=0 ) -break ; + result = entry->lle_comp_ops->lco_getattr (env, lov, index, + entry, &lov_attr); +if (result <0 ) +return result; index++; +if (!lov_attr) +continue ; +/* merge results */ attr->cat_blocks += lov_attr->cat_blocks;if (attr->cat_size < lov_attr->cat_size)@@ -679,7 +856,7 @@ static int lov_attr_get_composite(const struct lu_env *env,if (attr->cat_mtime < lov_attr->cat_mtime) attr->cat_mtime = lov_attr->cat_mtime; } -return result; +return 0 ; }static const struct lov_layout_operations lov_dispatch[] = {@@ -1235,6 +1412,49 @@ struct fiemap_state {bool fs_enough; }; +static struct cl_object *lov_find_subobj (const struct lu_env *env, +struct lov_object *lov, +struct lov_stripe_md *lsm, +int index) +{ +struct lov_device *dev =lu2lov_dev (lov2lu (lov)->lo_dev); +struct lov_thread_info *lti =lov_env_info (env); +struct lu_fid *ofid = <i->lti_fid; +struct lov_oinfo *oinfo; +struct cl_device *subdev; +int entry =lov_comp_entry (index); +int stripe =lov_comp_stripe (index); +int ost_idx; +int rc; +struct cl_object *result; + +if (lov->lo_type !=LLT_COMP ) { + result =NULL ; + goto out; + } + +if (entry >= lsm->lsm_entry_count || + stripe >= lsm->lsm_entries[entry]->lsme_stripe_count) { + result =NULL ; + goto out; + } + + oinfo = lsm->lsm_entries[entry]->lsme_oinfo[stripe]; + ost_idx = oinfo->loi_ost_idx; + rc =ostid_to_fid (ofid, &oinfo->loi_oi, ost_idx); +if (rc !=0 ) { + result =NULL ; + goto out; + } + + subdev =lovsub2cl_dev (dev->ld_target[ost_idx]); + result =lov_sub_find (env, subdev, ofid,NULL ); +out: +if (!result) + result =ERR_PTR (-EINVAL ); +return result; +} + static int fiemap_for_stripe(const struct lu_env *env, struct cl_object *obj, struct lov_stripe_md *lsm, struct fiemap *fiemap, size_t *buflen, struct ll_fiemap_info_key *fmkey,@@ -1457,6 +1677,12 @@ static int lov_object_fiemap(const struct lu_env *env, struct cl_object *obj, } } +/* No support for DOM layout yet. */ +if (lsme_is_dom (lsm->lsm_entries[0 ])) { + rc = -ENOTSUPP ; + goto out_lsm; + } +if (lsm->lsm_is_released) {if (fiemap->fm_start < fmkey->lfik_oa.o_size) {/** fs/lustre/lov/lov_offset.c +3
-0@@ -43,6 +43,9 @@ static u64 stripe_width(struct lov_stripe_md *lsm, unsigned int index)LASSERT (index < lsm->lsm_entry_count); +if (lsme_is_dom (entry)) +return (loff_t )entry->lsme_stripe_size; +return entry->lsme_stripe_size * entry->lsme_stripe_count; }fs/lustre/mdc/mdc_request.c +6
-1@@ -2265,7 +2265,12 @@ static int mdc_set_info_async(const struct lu_env *env,return 0 ; } -CERROR ("Unknown key %s\n" , (char *)key); +/* TODO: these OSC-related keys are ignored for now */ +if (KEY_IS (KEY_CHECKSUM ) ||KEY_IS (KEY_CACHE_SET ) || +KEY_IS (KEY_CACHE_LRU_SHRINK ) ||KEY_IS (KEY_GRANT_SHRINK )) +return 0 ; + +CERROR ("%s: Unknown key %s\n" , exp->exp_obd->obd_name, (char *)key);return -EINVAL ; }fs/lustre/obdclass/obd_config.c +35
-1@@ -972,7 +972,6 @@ int class_process_config(struct lustre_cfg *lcfg) err = -EINVAL ; goto out; } -switch (lcfg->lcfg_command) {case LCFG_SETUP : { err =class_setup (obd, lcfg);@@ -1020,6 +1019,41 @@ int class_process_config(struct lustre_cfg *lcfg) err =0 ; goto out; } +/* Process config log ADD_MDC record twice to add MDC also to LOV +* for Data-on-MDT: +* +* add 0:lustre-clilmv 1:lustre-MDT0000_UUID 2:0 3:1 +* 4:lustre-MDT0000-mdc_UUID +*/ +case LCFG_ADD_MDC : { +struct obd_device *lov_obd; +char *clilmv; + + err =obd_process_config (obd,sizeof (*lcfg), lcfg); +if (err) + goto out; + +/* make sure this is client LMV log entry */ + clilmv =strstr (lustre_cfg_string (lcfg,0 ),"clilmv" ); +if (!clilmv) + goto out; + +/* replace 'lmv' with 'lov' name to address LOV device and +* process llog record to add MDC there. +*/ + clilmv[4 ] ='o' ; + lov_obd =class_name2obd (lustre_cfg_string (lcfg,0 )); +if (!lov_obd) { + err = -ENOENT ; +CERROR ("%s: Cannot find LOV by %s name, rc = %d\n" , + obd->obd_name,lustre_cfg_string (lcfg,0 ), err); + }else { + err =obd_process_config (lov_obd,sizeof (*lcfg), lcfg); + } +/* restore 'lmv' name */ + clilmv[4 ] ='m' ; + goto out; + }default : { err =obd_process_config (obd,sizeof (*lcfg), lcfg); goto out;fs/lustre/ptlrpc/wiretest.c +2
-2@@ -1479,8 +1479,8 @@ void lustre_assert_wire_constants(void) (unsigned int)LOV_PATTERN_RAID0);LASSERTF (LOV_PATTERN_RAID1 ==0x00000002UL ,"found 0x%.8xUL\n" , (unsigned int )LOV_PATTERN_RAID1 ); -LASSERTF (LOV_PATTERN_FIRST ==0x00000100UL ,"found 0x%.8xUL\n" , - (unsigned int )LOV_PATTERN_FIRST ); +LASSERTF (LOV_PATTERN_MDT ==0x00000100UL ,"found 0x%.8xUL\n" , + (unsigned int )LOV_PATTERN_MDT );LASSERTF (LOV_PATTERN_CMOBD ==0x00000200UL ,"found 0x%.8xUL\n" , (unsigned int )LOV_PATTERN_CMOBD );include/uapi/linux/lustre/lustre_user.h +1
-1@@ -337,7 +337,7 @@ enum ll_lease_type {#define LOV_PATTERN_RAID0 0x001#define LOV_PATTERN_RAID1 0x002 -#define LOV_PATTERN_FIRST 0x100 +#define LOV_PATTERN_MDT 0x100#define LOV_PATTERN_CMOBD 0x200#define LOV_PATTERN_F_MASK 0xffff0000Signature · 1 line
1.8.3.1
PATCH 44/151 lustre: mdt: IO request handling in MDT
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Add methods to handle IO requests in MDT similar to OFD. Introduce MDS_INODEBITS_DOM bit for data on MDT, destinguish IO requests to MDT and OST and take appropriate lock in target code.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 2bcc5ad0ed6a ("LU-3285 mdt: IO request handling in MDT")
Reviewed-on: https://review.whamcloud.com/28013
--- fs/lustre/include/lustre_osc.h | 3 +++ fs/lustre/llite/llite_lib.c | 12 +++++++----- fs/lustre/mdc/mdc_request.c | 19 +++++++++++++++++-- fs/lustre/osc/osc_request.c | 3 ++- include/uapi/linux/lustre/lustre_idl.h | 5 +++-- 5 files changed, 32 insertions(+), 10 deletions(-)
fs/lustre/include/lustre_osc.h +3 -0
@@ -512,6 +512,9 @@ int osc_attr_update(const struct lu_env *env, struct cl_object *obj,int osc_object_glimpse (const struct lu_env *env,const struct cl_object *obj,struct ost_lvb *lvb); +/* osc_request.c */ +void osc_init_grant (struct client_obd *cli,struct obd_connect_data *ocd); +/**************************************************************************** * * Accessors and type conversions.
fs/lustre/llite/llite_lib.c +7 -5
@@ -187,7 +187,7 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)/* indicate MDT features supported by this client */ data->ocd_connect_flags =OBD_CONNECT_IBITS |OBD_CONNECT_NODEVOH | -OBD_CONNECT_ATTRFID | +OBD_CONNECT_ATTRFID |OBD_CONNECT_GRANT |OBD_CONNECT_VERSION |OBD_CONNECT_BRW_SIZE |OBD_CONNECT_CANCELSET |OBD_CONNECT_FID | OBD_CONNECT_AT | OBD_CONNECT_LOV_V3 |
@@ -201,10 +201,10 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)OBD_CONNECT_FLOCK_DEAD |OBD_CONNECT_DISP_STRIPE |OBD_CONNECT_LFSCK |OBD_CONNECT_OPEN_BY_FID | -OBD_CONNECT_DIR_STRIPE | -OBD_CONNECT_BULK_MBITS | -OBD_CONNECT_SUBTREE | -OBD_CONNECT_FLAGS2 |OBD_CONNECT_MULTIMODRPCS ; +OBD_CONNECT_DIR_STRIPE | +OBD_CONNECT_BULK_MBITS |OBD_CONNECT_CKSUM | +OBD_CONNECT_SUBTREE | +OBD_CONNECT_FLAGS2 |OBD_CONNECT_MULTIMODRPCS ; data->ocd_connect_flags2 =0 ;
@@ -215,6 +215,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)OBD_CONNECT_LARGE_ACL ;#endif + data->ocd_cksum_types =cksum_types_supported_client (); +if (OBD_FAIL_CHECK (OBD_FAIL_MDC_LIGHTWEIGHT ))/* flag mdc connection as lightweight, only used for test * purpose, use with care
fs/lustre/mdc/mdc_request.c +17 -2
@@ -54,6 +54,7 @@#include <uapi/linux/lustre/lustre_param.h> #include <lustre_swab.h> #include <obd_class.h> +#include <lustre_osc.h> #include "mdc_internal.h"
@@ -2354,6 +2355,15 @@ static int mdc_import_event(struct obd_device *obd, struct obd_import *imp,LASSERT (imp->imp_obd == obd);switch (event) { +case IMP_EVENT_DISCON : { +struct client_obd *cli = &obd->u.cli; + +spin_lock (&cli->cl_loi_list_lock); + cli->cl_avail_grant =0 ; + cli->cl_lost_grant =0 ; +spin_unlock (&cli->cl_loi_list_lock); +break ; + }case IMP_EVENT_INACTIVE : {struct client_obd *cli = &obd->u.cli;/*
@@ -2384,10 +2394,15 @@ static int mdc_import_event(struct obd_device *obd, struct obd_import *imp, mdc_hsm_ct_reregister, (void *)imp);break ; -case IMP_EVENT_OCD : +case IMP_EVENT_OCD : { +struct obd_connect_data *ocd = &imp->imp_connect_data; + +if (OCD_HAS_FLAG (ocd,GRANT )) +osc_init_grant (&obd->u.cli, ocd); + rc =obd_notify_observer (obd, obd,OBD_NOTIFY_OCD );break ; -case IMP_EVENT_DISCON : + }case IMP_EVENT_DEACTIVATE :case IMP_EVENT_ACTIVATE :break ;
fs/lustre/osc/osc_request.c +2 -1
@@ -897,7 +897,7 @@ static int osc_del_shrink_grant(struct client_obd *client) TIMEOUT_GRANT); } -static void osc_init_grant (struct client_obd *cli,struct obd_connect_data *ocd) +void osc_init_grant (struct client_obd *cli,struct obd_connect_data *ocd) {/* * ocd_grant is the total grant amount we're expect to hold: if we've
@@ -953,6 +953,7 @@ static void osc_init_grant(struct client_obd *cli, struct obd_connect_data *ocd) list_empty(&cli->cl_grant_shrink_list))osc_add_shrink_grant (cli); } +EXPORT_SYMBOL (osc_init_grant);/* We assume that the reason this OSC got a short read is because it read * beyond the end of a stripe file; i.e. lustre is reading a sparse file
include/uapi/linux/lustre/lustre_idl.h +3 -2
@@ -100,7 +100,7 @@#define MDC_REPLY_PORTAL 10/*#define MDC_BULK_PORTAL 11 */ #define MDS_REQUEST_PORTAL 12 -/*#define MDS_REPLY_PORTAL 13 */ +#define MDS_IO_PORTAL 13#define MDS_BULK_PORTAL 14#define LDLM_CB_REQUEST_PORTAL 15#define LDLM_CB_REPLY_PORTAL 16
@@ -1429,8 +1429,9 @@ enum mdt_reint_cmd {*/ #define MDS_INODELOCK_PERM 0x000010#define MDS_INODELOCK_XATTR 0x000020/* extended attributes */ +#define MDS_INODELOCK_DOM 0x000040/* Data for data-on-mdt files */ -#define MDS_INODELOCK_MAXSHIFT 5 +#define MDS_INODELOCK_MAXSHIFT 6/* This FULL lock is useful to take on unlink sort of operations */ #define MDS_INODELOCK_FULL ((1 << (MDS_INODELOCK_MAXSHIFT + 1)) - 1)
Signature · 1 line
1.8.3.1
PATCH 45/151 lustre: osc: common client setup/cleanup
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Make OSC code related to client setup/cleanup common for both OSB and MDC. Add new events in mdc_import_event().
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: a1bd0f62ee08 ("LU-3285 osc: common client setup/cleanup")
Reviewed-on: https://review.whamcloud.com/28014
--- fs/lustre/include/lustre_osc.h | 30 +++++++++++++-- fs/lustre/include/obd.h | 4 ++ fs/lustre/ldlm/ldlm_lib.c | 3 ++ fs/lustre/mdc/mdc_request.c | 59 +++++++++++++---------------- fs/lustre/obdclass/genops.c | 22 +++++------ fs/lustre/osc/osc_cache.c | 17 ++------- fs/lustre/osc/osc_internal.h | 1 + fs/lustre/osc/osc_object.c | 2 +- fs/lustre/osc/osc_request.c | 85 +++++++++++++++++++++++++++--------------- 9 files changed, 130 insertions(+), 93 deletions(-)
fs/lustre/include/lustre_osc.h +27 -3
@@ -51,6 +51,7 @@#include <obd.h> #include <cl_object.h> +#include <linux/libcfs/libcfs_hash.h> struct osc_quota_info {/* linkage for quota hash table */
@@ -479,9 +480,23 @@ int osc_cache_writeback_range(const struct lu_env *env, struct osc_object *obj, pgoff_t start, pgoff_t end, int hp, int discard);int osc_cache_wait_range (const struct lu_env *env,struct osc_object *obj,pgoff_t start,pgoff_t end); -void osc_io_unplug (const struct lu_env *env,struct client_obd *cli, -struct osc_object *osc); -int lru_queue_work (const struct lu_env *env,void *data); + +int __osc_io_unplug (const struct lu_env *env,struct client_obd *cli, +struct osc_object *osc,int async); + +static inline int osc_io_unplug_async (const struct lu_env *env, +struct client_obd *cli, +struct osc_object *osc) +{ +return __osc_io_unplug (env, cli, osc,1 ); +} + +static inline void osc_io_unplug (const struct lu_env *env, +struct client_obd *cli, +struct osc_object *osc) +{ + (void )__osc_io_unplug (env, cli, osc,0 ); +}void osc_object_set_contended (struct osc_object *obj);void osc_object_clear_contended (struct osc_object *obj);
@@ -511,9 +526,18 @@ int osc_attr_update(const struct lu_env *env, struct cl_object *obj, const struct cl_attr *attr, unsigned int valid);int osc_object_glimpse (const struct lu_env *env,const struct cl_object *obj,struct ost_lvb *lvb); +int osc_object_invalidate (const struct lu_env *env,struct osc_object *osc);/* osc_request.c */ void osc_init_grant (struct client_obd *cli,struct obd_connect_data *ocd); +int osc_setup_common (struct obd_device *obd,struct lustre_cfg *lcfg); +int osc_precleanup_common (struct obd_device *obd); +int osc_cleanup_common (struct obd_device *obd); +int osc_set_info_async (const struct lu_env *env,struct obd_export *exp, +u32 keylen,void *key,u32 vallen,void *val, +struct ptlrpc_request_set *set); +int osc_ldlm_resource_invalidate (struct cfs_hash *hs,struct cfs_hash_bd *bd, +struct hlist_node *hnode,void *arg);/**************************************************************************** *
fs/lustre/include/obd.h +4 -0
@@ -316,6 +316,10 @@ struct client_obd {atomic_t cl_mgc_refcount;struct obd_export *cl_mgc_mgsexp; +/* in-flight control list and total RPCs counter */ +struct list_head cl_flight_waiters; +u32 cl_rpcs_in_flight; +/* checksumming for data sent over the network */ unsigned int cl_checksum:1, /* 0 = disabled, 1 = enabled */ cl_checksum_dump:1; /* same */
fs/lustre/ldlm/ldlm_lib.c +3 -0
@@ -351,6 +351,9 @@ int client_obd_setup(struct obd_device *obddev, struct lustre_cfg *lcfg)atomic_long_set (&cli->cl_unstable_count,0 );INIT_LIST_HEAD (&cli->cl_shrink_list); +INIT_LIST_HEAD (&cli->cl_flight_waiters); + cli->cl_rpcs_in_flight =0 ; +init_waitqueue_head (&cli->cl_destroy_waitq);atomic_set (&cli->cl_destroy_in_flight,0 );/* Turn on checksumming by default. */
fs/lustre/mdc/mdc_request.c +26 -33
131 lines not on this page · raw message
fs/lustre/obdclass/genops.c +11 -11
@@ -1291,14 +1291,14 @@ int obd_get_request_slot(struct client_obd *cli)int rc;spin_lock (&cli->cl_loi_list_lock); -if (cli->cl_r_in_flight < cli->cl_max_rpcs_in_flight) { - cli->cl_r_in_flight++; +if (cli->cl_rpcs_in_flight < cli->cl_max_rpcs_in_flight) { + cli->cl_rpcs_in_flight++;spin_unlock (&cli->cl_loi_list_lock);return 0 ; }init_waitqueue_head (&orsw.orsw_waitq); -list_add_tail (&orsw.orsw_entry, &cli->cl_loi_read_list); +list_add_tail (&orsw.orsw_entry, &cli->cl_flight_waiters); orsw.orsw_signaled = false;spin_unlock (&cli->cl_loi_list_lock);
@@ -1314,7 +1314,7 @@ int obd_get_request_slot(struct client_obd *cli)if (rc) {if (!orsw.orsw_signaled) {if (list_empty (&orsw.orsw_entry)) - cli->cl_r_in_flight--; + cli->cl_rpcs_in_flight--;else list_del (&orsw.orsw_entry); }
@@ -1336,16 +1336,16 @@ void obd_put_request_slot(struct client_obd *cli)struct obd_request_slot_waiter *orsw;spin_lock (&cli->cl_loi_list_lock); - cli->cl_r_in_flight--; + cli->cl_rpcs_in_flight--;/* If there is free slot, wakeup the first waiter. */ -if (!list_empty (&cli->cl_loi_read_list) && -likely (cli->cl_r_in_flight < cli->cl_max_rpcs_in_flight)) { - orsw =list_first_entry (&cli->cl_loi_read_list, +if (!list_empty (&cli->cl_flight_waiters) && +likely (cli->cl_rpcs_in_flight < cli->cl_max_rpcs_in_flight)) { + orsw =list_first_entry (&cli->cl_flight_waiters, struct obd_request_slot_waiter, orsw_entry);list_del_init (&orsw->orsw_entry); - cli->cl_r_in_flight++; + cli->cl_rpcs_in_flight++;wake_up (&orsw->orsw_waitq); }spin_unlock (&cli->cl_loi_list_lock);
@@ -1395,14 +1395,14 @@ int obd_set_max_rpcs_in_flight(struct client_obd *cli, u32 max)/* We increase the max_rpcs_in_flight, then wakeup some waiters. */ for (i =0 ; i < diff; i++) { - orsw =list_first_entry_or_null (&cli->cl_loi_read_list, + orsw =list_first_entry_or_null (&cli->cl_flight_waiters, struct obd_request_slot_waiter, orsw_entry);if (!orsw)break ;list_del_init (&orsw->orsw_entry); - cli->cl_r_in_flight++; + cli->cl_rpcs_in_flight++;wake_up (&orsw->orsw_waitq); }spin_unlock (&cli->cl_loi_list_lock);
fs/lustre/osc/osc_cache.c +3 -14
@@ -2260,8 +2260,8 @@ static void osc_check_rpcs(const struct lu_env *env, struct client_obd *cli)spin_unlock (&cli->cl_loi_list_lock); } -static int __osc_io_unplug (const struct lu_env *env,struct client_obd *cli, -struct osc_object *osc,int async) +int __osc_io_unplug (const struct lu_env *env,struct client_obd *cli, +struct osc_object *osc,int async) {int rc =0 ;
@@ -2278,18 +2278,7 @@ static int __osc_io_unplug(const struct lu_env *env, struct client_obd *cli,return rc; } - -static int osc_io_unplug_async (const struct lu_env *env, -struct client_obd *cli,struct osc_object *osc) -{ -return __osc_io_unplug (env, cli, osc,1 ); -} - -void osc_io_unplug (const struct lu_env *env,struct client_obd *cli, -struct osc_object *osc) -{ - (void )__osc_io_unplug (env, cli, osc,0 ); -} +EXPORT_SYMBOL (__osc_io_unplug);int osc_prep_async_page (struct osc_object *osc,struct osc_page *ops,struct page *page,loff_t offset)
fs/lustre/osc/osc_internal.h +1 -0
@@ -44,6 +44,7 @@int osc_shrink_grant_to_target (struct client_obd *cli,u64 target_bytes);void osc_update_next_shrink (struct client_obd *cli); +int lru_queue_work (const struct lu_env *env,void *data);typedef int (*osc_enqueue_upcall_f )(void *cookie,struct lustre_handle *lockh,int rc);
fs/lustre/osc/osc_object.c +1 -1
@@ -462,5 +462,5 @@ int osc_object_invalidate(const struct lu_env *env, struct osc_object *osc)return 0 ; } - +EXPORT_SYMBOL (osc_object_invalidate);/** @} osc */
fs/lustre/osc/osc_request.c +54 -31
169 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH 46/151 lustre: mdc: add IO methods to the MDC
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Add CL IO methods specific for MDC. Update and export related OSC functions to be used from MDC.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 59bd024b7117 ("LU-3285 mdc: add IO methods to the MDC")
Reviewed-on: https://review.whamcloud.com/28015
--- fs/lustre/include/lustre_osc.h | 39 +++++++++-- fs/lustre/mdc/mdc_dev.c | 155 ++++++++++++++++++++++++++++++++++++++--- fs/lustre/osc/osc_cache.c | 4 ++ fs/lustre/osc/osc_dev.c | 2 + fs/lustre/osc/osc_internal.h | 13 +++- fs/lustre/osc/osc_io.c | 76 +++++++++++--------- fs/lustre/osc/osc_page.c | 1 + fs/lustre/osc/osc_request.c | 25 ++++--- 8 files changed, 253 insertions(+), 62 deletions(-)
fs/lustre/include/lustre_osc.h +35 -4
@@ -447,18 +447,19 @@ struct osc_page {#define OSC_FLAGS (ASYNC_URGENT | ASYNC_READY) +/* osc_page.c */ int osc_page_init (const struct lu_env *env,struct cl_object *obj,struct cl_page *page,pgoff_t ind);void osc_index2policy (union ldlm_policy_data *policy,const struct cl_object *obj,pgoff_t start,pgoff_t end); -int osc_lvb_print (const struct lu_env *env,void *cookie, -lu_printer_t p,const struct ost_lvb *lvb); -void osc_lru_add_batch (struct client_obd *cli,struct list_head *list);void osc_page_submit (const struct lu_env *env,struct osc_page *opg,enum cl_req_type crt,int brw_flags); +int lru_queue_work (const struct lu_env *env,void *data); + +/* osc_cache.c */ int osc_set_async_flags (struct osc_object *obj,struct osc_page *opg,u32 async_flags); int osc_prep_async_page(struct osc_object *osc, struct osc_page *ops,
@@ -480,7 +481,6 @@ int osc_cache_writeback_range(const struct lu_env *env, struct osc_object *obj, pgoff_t start, pgoff_t end, int hp, int discard);int osc_cache_wait_range (const struct lu_env *env,struct osc_object *obj,pgoff_t start,pgoff_t end); -int __osc_io_unplug (const struct lu_env *env,struct client_obd *cli,struct osc_object *osc,int async);
@@ -539,6 +539,37 @@ int osc_set_info_async(const struct lu_env *env, struct obd_export *exp,int osc_ldlm_resource_invalidate (struct cfs_hash *hs,struct cfs_hash_bd *bd,struct hlist_node *hnode,void *arg); +int osc_punch_send (struct obd_export *exp,struct obdo *oa, +obd_enqueue_update_f upcall,void *cookie); + +/* osc_io.c */ +int osc_io_submit (const struct lu_env *env,const struct cl_io_slice *ios, +enum cl_req_type crt,struct cl_2queue *queue); +int osc_io_commit_async (const struct lu_env *env, +const struct cl_io_slice *ios, +struct cl_page_list *qin,int from,int to, +cl_commit_cbt cb); +int osc_io_iter_init (const struct lu_env *env,const struct cl_io_slice *ios); +void osc_io_iter_fini (const struct lu_env *env, +const struct cl_io_slice *ios); +int osc_io_write_iter_init (const struct lu_env *env, +const struct cl_io_slice *ios); +void osc_io_write_iter_fini (const struct lu_env *env, +const struct cl_io_slice *ios); +int osc_io_fault_start (const struct lu_env *env,const struct cl_io_slice *ios); +void osc_io_setattr_end (const struct lu_env *env, +const struct cl_io_slice *slice); +int osc_io_read_start (const struct lu_env *env, +const struct cl_io_slice *slice); +int osc_io_write_start (const struct lu_env *env, +const struct cl_io_slice *slice); +void osc_io_end (const struct lu_env *env,const struct cl_io_slice *slice); + +int osc_io_fsync_start (const struct lu_env *env, +const struct cl_io_slice *slice); +void osc_io_fsync_end (const struct lu_env *env, +const struct cl_io_slice *slice); +/**************************************************************************** * * Accessors and type conversions.
fs/lustre/mdc/mdc_dev.c +144 -11
174 lines not on this page · raw message
fs/lustre/osc/osc_cache.c +4 -0
@@ -2309,6 +2309,7 @@ int osc_prep_async_page(struct osc_object *osc, struct osc_page *ops,return 0 ; } +EXPORT_SYMBOL (osc_prep_async_page);int osc_queue_async_io (const struct lu_env *env,struct cl_io *io,struct osc_page *ops)
@@ -2798,6 +2799,7 @@ int osc_cache_truncate_start(const struct lu_env *env, struct osc_object *obj,return result; } +EXPORT_SYMBOL (osc_cache_truncate_start);/** * Called after osc_io_setattr_end to add oio->oi_trunc back to cache.
@@ -2883,6 +2885,7 @@ int osc_cache_wait_range(const struct lu_env *env, struct osc_object *obj,OSC_IO_DEBUG (obj,"sync file range.\n" );return result; } +EXPORT_SYMBOL (osc_cache_wait_range);/** * Called to write out a range of osc object.
@@ -3004,6 +3007,7 @@ int osc_cache_writeback_range(const struct lu_env *env, struct osc_object *obj,OSC_IO_DEBUG (obj,"pageout [%lu, %lu], %d.\n" , start, end, result);return result; } +EXPORT_SYMBOL (osc_cache_writeback_range);/** * Returns a list of pages by a given [start, end] of @obj.
fs/lustre/osc/osc_dev.c +2 -0
@@ -126,6 +126,7 @@ struct lu_context_key osc_key = { .lct_init = osc_key_init, .lct_fini = osc_key_fini }; +EXPORT_SYMBOL (osc_key);static void *osc_session_init (const struct lu_context *ctx,struct lu_context_key *key)
@@ -151,6 +152,7 @@ struct lu_context_key osc_session_key = { .lct_init = osc_session_init, .lct_fini = osc_session_fini }; +EXPORT_SYMBOL (osc_session_key);/* type constructor/destructor: osc_type_{init,fini,start,stop}(). */ LU_TYPE_INIT_FINI (osc, &osc_key, &osc_session_key);
fs/lustre/osc/osc_internal.h +10 -3
@@ -65,9 +65,6 @@ int osc_match_base(struct obd_export *exp, struct ldlm_res_id *res_id,int osc_setattr_async (struct obd_export *exp,struct obdo *oa,obd_enqueue_update_f upcall,void *cookie,struct ptlrpc_request_set *rqset); -int osc_punch_base (struct obd_export *exp,struct obdo *oa, -obd_enqueue_update_f upcall,void *cookie, -struct ptlrpc_request_set *rqset);int osc_sync_base (struct osc_object *exp,struct obdo *oa,obd_enqueue_update_f upcall,void *cookie,struct ptlrpc_request_set *rqset);
@@ -169,4 +166,14 @@ unsigned long osc_cache_shrink_count(struct shrinker *sk,unsigned long osc_cache_shrink_scan (struct shrinker *sk,struct shrink_control *sc); +static inline void osc_set_io_portal (struct ptlrpc_request *req) +{ +struct obd_import *imp = req->rq_import; + +/* Distinguish OSC from MDC here to use OST or MDS portal */ +if (OCD_HAS_FLAG (&imp->imp_connect_data,IBITS )) + req->rq_request_portal =MDS_IO_PORTAL ; +else + req->rq_request_portal =OST_IO_PORTAL ; +}#endif /* OSC_INTERNAL_H */
fs/lustre/osc/osc_io.c +42 -34
205 lines not on this page · raw message
fs/lustre/osc/osc_page.c +1 -0
@@ -276,6 +276,7 @@ int osc_page_init(const struct lu_env *env, struct cl_object *obj,return result; } +EXPORT_SYMBOL (osc_page_init);/** * Helper function called by osc_io_submit() for every page in an immediate
fs/lustre/osc/osc_request.c +15 -10
@@ -410,30 +410,32 @@ static int osc_create(const struct lu_env *env, struct obd_export *exp,return rc; } -int osc_punch_base (struct obd_export *exp,struct obdo *oa, -obd_enqueue_update_f upcall,void *cookie, -struct ptlrpc_request_set *rqset) +int osc_punch_send (struct obd_export *exp,struct obdo *oa, +obd_enqueue_update_f upcall,void *cookie) {struct ptlrpc_request *req;struct osc_setattr_args *sa; +struct obd_import *imp =class_exp2cliimp (exp);struct ost_body *body;int rc; - req =ptlrpc_request_alloc (class_exp2cliimp (exp), &RQF_OST_PUNCH ); + req =ptlrpc_request_alloc (imp, &RQF_OST_PUNCH );if (!req)return -ENOMEM ; rc =ptlrpc_request_pack (req,LUSTRE_OST_VERSION ,OST_PUNCH ); -if (rc) { +if (rc <0 ) {ptlrpc_request_free (req);return rc; } - req->rq_request_portal =OST_IO_PORTAL ;/* bug 7198 */ + +osc_set_io_portal (req); +ptlrpc_at_set_req_timeout (req); body =req_capsule_client_get (&req->rq_pill, &RMF_OST_BODY ); -LASSERT (body); -lustre_set_wire_obdo (&req->rq_import->imp_connect_data, &body->oa, + +lustre_set_wire_obdo (&imp->imp_connect_data, &body->oa, oa);ptlrpc_request_set_replen (req);
@@ -444,10 +446,12 @@ int osc_punch_base(struct obd_export *exp, struct obdo *oa, sa->sa_oa = oa; sa->sa_upcall = upcall; sa->sa_cookie = cookie; -ptlrpc_set_add_req (rqset, req); + +ptlrpcd_add_req (req);return 0 ; } +EXPORT_SYMBOL (osc_punch_send); static int osc_sync_interpret(const struct lu_env *env, struct ptlrpc_request *req,
@@ -1157,7 +1161,8 @@ static int osc_brw_prep_request(int cmd, struct client_obd *cli,ptlrpc_request_free (req);return rc; } - req->rq_request_portal =OST_IO_PORTAL ;/* bug 7198 */ +osc_set_io_portal (req); +ptlrpc_at_set_req_timeout (req);/* ask ptlrpc not to resend on EINPROGRESS since BRWs have their own * retry logic
Signature · 1 line
1.8.3.1
PATCH 47/151 lustre: lvbo: pass lock as parameter to lvbo_update()
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
LVBO lvbo_update call was using ldlm_resource as parameter, the ldlm_lock parameter is added to recognize DoM resource in MDT LVBO functions and perform DoM specific actions.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 978428ed0476 ("LU-3285 lvbo: pass lock as parameter to lvbo_update()")
Reviewed-on: https://review.whamcloud.com/28016
--- fs/lustre/include/lustre_dlm.h | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
fs/lustre/include/lustre_dlm.h +16 -10
@@ -279,9 +279,8 @@ struct ldlm_pool {*/ struct ldlm_valblock_ops {int (*lvbo_init)(struct ldlm_resource *res); -int (*lvbo_update)(struct ldlm_resource *res, -struct ptlrpc_request *r, -int increase); +int (*lvbo_update)(struct ldlm_resource *res,struct ldlm_lock *lock, +struct ptlrpc_request *r,int increase);int (*lvbo_free)(struct ldlm_resource *res);/* Return size of lvb data appropriate RPC size can be reserved */ int (*lvbo_size)(struct ldlm_lock *lock);
@@ -1094,17 +1093,24 @@ static inline struct ldlm_lock *ldlm_handle2lock(const struct lustre_handle *h)* Update Lock Value Block Operations (LVBO) on a resource taking into account * data from request @r */ -static inline int ldlm_res_lvbo_update (struct ldlm_resource *res, -struct ptlrpc_request *r,int increase) +static inline int ldlm_lvbo_update (struct ldlm_resource *res, +struct ldlm_lock *lock, +struct ptlrpc_request *req,int increase) { -if (ldlm_res_to_ns (res)->ns_lvbo && -ldlm_res_to_ns (res)->ns_lvbo->lvbo_update) { -return ldlm_res_to_ns (res)->ns_lvbo->lvbo_update (res, r, - increase); - } +struct ldlm_namespace *ns =ldlm_res_to_ns (res); + +if (ns->ns_lvbo && ns->ns_lvbo->lvbo_update) +return ns->ns_lvbo->lvbo_update (res, lock, req, increase); +return 0 ; } +static inline int ldlm_res_lvbo_update (struct ldlm_resource *res, +struct ptlrpc_request *req,int increase) +{ +return ldlm_lvbo_update (res,NULL , req, increase); +} +int ldlm_error2errno (enum ldlm_error error);#if LUSTRE_TRACKS_LOCK_EXP_REFS
Signature · 1 line
1.8.3.1
PATCH 48/151 lustre: mds: add IO locking to the MDC and MDT
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
- introduce new DOM inodebit for Data-on-MDT files.
- add IO lock and glimpse handling at MDT along with needed LVB updates for it.
- a MDC is updated to exclude DOM bit from ELC and to handle LVB changes due to glimpse on MDT.
- add CLIO locking at MDC, it uses IBITS lock to protect data at MDT and a MDC handles such locks to convert them into proper CLIO locks.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 4f35c341f22b ("LU-3285 mds: add IO locking to the MDC and MDT")
Reviewed-on: https://review.whamcloud.com/28018
Lustre-commit: 138b3eee3b61 ("LU-3285 osc: remove wrongly applied assertion.")
Reviewed-on: https://review.whamcloud.com/29811
--- fs/lustre/include/lustre_dlm.h | 11 +- fs/lustre/include/lustre_dlm_flags.h | 2 +- fs/lustre/include/lustre_osc.h | 147 ++++- fs/lustre/ldlm/ldlm_internal.h | 1 + fs/lustre/ldlm/ldlm_request.c | 4 +- fs/lustre/llite/file.c | 5 +- fs/lustre/llite/llite_lib.c | 1 + fs/lustre/lmv/lmv_obd.c | 6 +- fs/lustre/mdc/mdc_dev.c | 960 ++++++++++++++++++++++++++++++++- fs/lustre/mdc/mdc_internal.h | 2 + fs/lustre/mdc/mdc_reint.c | 5 +- fs/lustre/osc/osc_cache.c | 12 +- fs/lustre/osc/osc_internal.h | 30 +- fs/lustre/osc/osc_io.c | 3 +- fs/lustre/osc/osc_lock.c | 122 ++--- fs/lustre/osc/osc_object.c | 31 +- fs/lustre/osc/osc_request.c | 25 +- include/uapi/linux/lustre/lustre_idl.h | 4 + 18 files changed, 1201 insertions(+), 170 deletions(-)
fs/lustre/include/lustre_dlm.h +10 -1
@@ -723,7 +723,10 @@ struct ldlm_lock {/** Private storage for lock user. Opaque to LDLM. */ void *l_ast_data; - +/* Separate ost_lvb used mostly by Data-on-MDT for now. +* It is introduced to don't mix with layout lock data. +*/ +struct ost_lvb l_ost_lvb;/* * Server-side-only members. */
@@ -870,6 +873,12 @@ static inline bool ldlm_has_layout(struct ldlm_lock *lock) lock->l_policy_data.l_inodebits.bits &MDS_INODELOCK_LAYOUT ; } +static inline bool ldlm_has_dom (struct ldlm_lock *lock) +{ +return lock->l_resource->lr_type ==LDLM_IBITS && + lock->l_policy_data.l_inodebits.bits &MDS_INODELOCK_DOM ; +} +static inline char *ldlm_ns_name (struct ldlm_namespace *ns) {
fs/lustre/include/lustre_dlm_flags.h +1 -1
@@ -393,7 +393,7 @@/** l_flags bits marked as "ast" bits */ #define LDLM_FL_AST_MASK (LDLM_FL_FLOCK_DEADLOCK |\ - LDLM_FL_AST_DISCARD_DATA) + LDLM_FL_DISCARD_DATA)/** l_flags bits marked as "blocked" bits */ #define LDLM_FL_BLOCKED_MASK (LDLM_FL_BLOCK_GRANTED |\
fs/lustre/include/lustre_osc.h +129 -18
209 lines not on this page · raw message
fs/lustre/ldlm/ldlm_internal.h +1 -0
@@ -37,6 +37,7 @@extern struct list_head ldlm_srv_namespace_list;extern struct mutex ldlm_cli_namespace_lock;extern struct list_head ldlm_cli_active_namespace_list; +extern struct kmem_cache *ldlm_glimpse_work_kmem;static inline int ldlm_namespace_nr_read (enum ldlm_side client) {
fs/lustre/ldlm/ldlm_request.c +2 -2
@@ -1488,8 +1488,8 @@ static int ldlm_prepare_lru_list(struct ldlm_namespace *ns, lock->l_flags |=LDLM_FL_CBPENDING |LDLM_FL_CANCELING ;if ((flags &LDLM_LRU_FLAG_CLEANUP ) && - lock->l_resource->lr_type ==LDLM_EXTENT && - lock->l_granted_mode ==LCK_PR ) + (lock->l_resource->lr_type ==LDLM_EXTENT || +ldlm_has_dom (lock)) && lock->l_granted_mode ==LCK_PR )ldlm_set_discard_data (lock);/* We can't re-add to l_lru as it confuses the
fs/lustre/llite/file.c +4 -1
@@ -1054,8 +1054,11 @@ int ll_merge_attr(const struct lu_env *env, struct inode *inode) rc =cl_object_attr_get (env, obj, attr);cl_object_attr_unlock (obj); -if (rc !=0 ) +if (rc !=0 ) { +if (rc == -ENODATA ) + rc =0 ; goto out_size_unlock; + }if (atime < attr->cat_atime) atime = attr->cat_atime;
fs/lustre/llite/llite_lib.c +1 -0
@@ -189,6 +189,7 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt) data->ocd_connect_flags =OBD_CONNECT_IBITS |OBD_CONNECT_NODEVOH |OBD_CONNECT_ATTRFID |OBD_CONNECT_GRANT |OBD_CONNECT_VERSION |OBD_CONNECT_BRW_SIZE | +OBD_CONNECT_SRVLOCK |OBD_CONNECT_TRUNCLOCK |OBD_CONNECT_CANCELSET |OBD_CONNECT_FID |OBD_CONNECT_AT |OBD_CONNECT_LOV_V3 | OBD_CONNECT_VBR | OBD_CONNECT_FULL20 |
fs/lustre/lmv/lmv_obd.c +3 -3
@@ -1914,7 +1914,7 @@ static int lmv_rename(struct obd_export *exp, struct md_op_data *op_data,return rc; rc =lmv_early_cancel (exp,NULL , op_data, src_tgt->ltd_idx, -LCK_EX ,MDS_INODELOCK_FULL , +LCK_EX ,MDS_INODELOCK_ELC ,MF_MDC_CANCEL_FID3 );if (rc)return rc;
@@ -1928,7 +1928,7 @@ static int lmv_rename(struct obd_export *exp, struct md_op_data *op_data,struct lmv_tgt_desc *tgt; rc =lmv_early_cancel (exp,NULL , op_data, src_tgt->ltd_idx, -LCK_EX ,MDS_INODELOCK_FULL , +LCK_EX ,MDS_INODELOCK_ELC ,MF_MDC_CANCEL_FID4 );if (rc)return rc;
@@ -2472,7 +2472,7 @@ static int lmv_unlink(struct obd_export *exp, struct md_op_data *op_data, } rc =lmv_early_cancel (exp,NULL , op_data, tgt->ltd_idx,LCK_EX , -MDS_INODELOCK_FULL ,MF_MDC_CANCEL_FID3 ); +MDS_INODELOCK_ELC ,MF_MDC_CANCEL_FID3 );if (rc !=0 )return rc;
fs/lustre/mdc/mdc_dev.c +956 -4
1,006 lines not on this page · raw message
fs/lustre/mdc/mdc_internal.h +2 -0
@@ -154,5 +154,7 @@ static inline unsigned long hash_x_index(u64 hash, int hash64)/* mdc_dev.c */ extern struct lu_device_type mdc_device_type; +int mdc_ldlm_blocking_ast (struct ldlm_lock *dlmlock, +struct ldlm_lock_desc *new,void *data,int flag);#endif
fs/lustre/mdc/mdc_reint.c +3 -2
@@ -270,9 +270,10 @@ int mdc_unlink(struct obd_export *exp, struct md_op_data *op_data, MDS_INODELOCK_UPDATE);if ((op_data->op_flags &MF_MDC_CANCEL_FID3 ) && (fid_is_sane (&op_data->op_fid3))) +/* don't cancel DoM lock which may cause data flush */ count +=mdc_resource_get_unused (exp, &op_data->op_fid3, &cancels,LCK_EX , -MDS_INODELOCK_FULL ); +MDS_INODELOCK_ELC ); req =ptlrpc_request_alloc (class_exp2cliimp (exp), &RQF_MDS_REINT_UNLINK );if (!req) {
@@ -373,7 +374,7 @@ int mdc_rename(struct obd_export *exp, struct md_op_data *op_data, (fid_is_sane(&op_data->op_fid4))) count +=mdc_resource_get_unused (exp, &op_data->op_fid4, &cancels,LCK_EX , -MDS_INODELOCK_FULL ); +MDS_INODELOCK_ELC ); req = ptlrpc_request_alloc(class_exp2cliimp(exp), op_data->op_cli_flags & CLI_MIGRATE ?
fs/lustre/osc/osc_cache.c +8 -4
@@ -248,7 +248,9 @@ static int __osc_extent_sanity_check(struct osc_extent *ext, goto out; } -if (ext->oe_dlmlock && !ldlm_is_failed (ext->oe_dlmlock)) { +if (ext->oe_dlmlock && + ext->oe_dlmlock->l_resource->lr_type ==LDLM_EXTENT && + !ldlm_is_failed (ext->oe_dlmlock)) {struct ldlm_extent *extent; extent = &ext->oe_dlmlock->l_policy_data.l_extent;
@@ -3096,6 +3098,7 @@ bool osc_page_gang_lookup(const struct lu_env *env, struct cl_io *io,return res; } +EXPORT_SYMBOL (osc_page_gang_lookup);/** * Check if page @page is covered by an extra lock or discard it.
@@ -3140,8 +3143,8 @@ static bool check_and_discard_cb(const struct lu_env *env, struct cl_io *io,return true; } -static bool discard_cb (const struct lu_env *env,struct cl_io *io, -struct osc_page *ops,void *cbdata) +bool osc_discard_cb (const struct lu_env *env,struct cl_io *io, +struct osc_page *ops,void *cbdata) {struct osc_thread_info *info =osc_env_info (env);struct cl_page *page = ops->ops_cl.cpl_page;
@@ -3163,6 +3166,7 @@ static bool discard_cb(const struct lu_env *env, struct cl_io *io,return true; } +EXPORT_SYMBOL (osc_discard_cb);/** * Discard pages protected by the given lock. This function traverses radix
@@ -3186,7 +3190,7 @@ int osc_lock_discard_pages(const struct lu_env *env, struct osc_object *osc,if (result !=0 ) goto out; - cb = discard ? discard_cb : check_and_discard_cb; + cb = discard ? osc_discard_cb : check_and_discard_cb; info->oti_fn_index = start; info->oti_next_index = start;
fs/lustre/osc/osc_internal.h +9 -21
@@ -45,9 +45,11 @@int osc_shrink_grant_to_target (struct client_obd *cli,u64 target_bytes);void osc_update_next_shrink (struct client_obd *cli);int lru_queue_work (const struct lu_env *env,void *data); - -typedef int (*osc_enqueue_upcall_f )(void *cookie,struct lustre_handle *lockh, -int rc); +int osc_extent_finish (const struct lu_env *env,struct osc_extent *ext, +int sent,int rc); +void osc_extent_release (const struct lu_env *env,struct osc_extent *ext); +int osc_lock_discard_pages (const struct lu_env *env,struct osc_object *osc, +pgoff_t start,pgoff_t end,bool discard); int osc_enqueue_base(struct obd_export *exp, struct ldlm_res_id *res_id, u64 *flags, union ldlm_policy_data *policy,
@@ -136,24 +138,10 @@ int osc_quotactl(struct obd_device *unused, struct obd_export *exp,void osc_dec_unstable_pages (struct ptlrpc_request *req);bool osc_over_unstable_soft_limit (struct client_obd *cli); -/* -* Bit flags for osc_dlm_lock_at_pageoff(). -*/ -enum osc_dap_flags { -/* -* Just check if the desired lock exists, it won't hold reference -* count on lock. -*/ -OSC_DAP_FL_TEST_LOCK =BIT (0 ), -/* -* Return the lock even if it is being canceled. -*/ -OSC_DAP_FL_CANCELING =BIT (1 ), -}; - -struct ldlm_lock *osc_dlmlock_at_pgoff (const struct lu_env *env, -struct osc_object *obj,pgoff_t index, -enum osc_dap_flags flags); +struct ldlm_lock *osc_obj_dlmlock_at_pgoff (const struct lu_env *env, +struct osc_object *obj, +pgoff_t index, +enum osc_dap_flags flags);int osc_object_invalidate (const struct lu_env *env,struct osc_object *osc);
fs/lustre/osc/osc_io.c +2 -1
@@ -57,7 +57,7 @@ static void osc_io_fini(const struct lu_env *env, const struct cl_io_slice *io) { } -static void osc_read_ahead_release (const struct lu_env *env,void *cbdata) +void osc_read_ahead_release (const struct lu_env *env,void *cbdata) {struct ldlm_lock *dlmlock = cbdata;struct lustre_handle lockh;
@@ -66,6 +66,7 @@ static void osc_read_ahead_release(const struct lu_env *env, void *cbdata)ldlm_lock_decref (&lockh,LCK_PR );LDLM_LOCK_PUT (dlmlock); } +EXPORT_SYMBOL (osc_read_ahead_release); static int osc_io_read_ahead(const struct lu_env *env, const struct cl_io_slice *ios,
fs/lustre/osc/osc_lock.c +43 -79
254 lines not on this page · raw message
fs/lustre/osc/osc_object.c +17 -14
@@ -50,6 +50,16 @@* Object operations. * */ +static void osc_obj_build_res_name (struct osc_object *osc, +struct ldlm_res_id *resname) +{ +ostid_build_res_name (&osc->oo_oinfo->loi_oi, resname); +} + +static const struct osc_object_operations osc_object_ops = { + .oto_build_res_name = osc_obj_build_res_name, + .oto_dlmlock_at_pgoff = osc_obj_dlmlock_at_pgoff, +};int osc_object_init (const struct lu_env *env,struct lu_object *obj,const struct lu_object_conf *conf)
@@ -78,6 +88,8 @@ int osc_object_init(const struct lu_env *env, struct lu_object *obj,spin_lock_init (&osc->oo_ol_spin);INIT_LIST_HEAD (&osc->oo_ol_list); +LASSERT (osc->oo_obj_ops); +cl_object_page_init (lu2cl (obj),sizeof (struct osc_page ));return 0 ;
@@ -189,7 +201,7 @@ static int osc_object_ast_clear(struct ldlm_lock *lock, void *data)return LDLM_ITER_CONTINUE ; } -static int osc_object_prune (const struct lu_env *env,struct cl_object *obj) +int osc_object_prune (const struct lu_env *env,struct cl_object *obj) {struct osc_object *osc =cl2osc (obj);struct ldlm_res_id *resname = &osc_env_info (env)->oti_resname;
@@ -197,11 +209,12 @@ static int osc_object_prune(const struct lu_env *env, struct cl_object *obj)/* DLM locks don't hold a reference of osc_object so we have to * clear it before the object is being destroyed. */ -ostid_build_res_name (&osc->oo_oinfo->loi_oi, resname); +osc_build_res_name (osc, resname);ldlm_resource_iterate (osc_export (osc)->exp_obd->obd_namespace, resname, osc_object_ast_clear, osc);return 0 ; } +EXPORT_SYMBOL (osc_object_prune); static int osc_object_fiemap(const struct lu_env *env, struct cl_object *obj, struct ll_fiemap_info_key *fmkey,
@@ -291,18 +304,6 @@ static int osc_object_fiemap(const struct lu_env *env, struct cl_object *obj,return rc; } -void osc_object_set_contended (struct osc_object *obj) -{ - obj->oo_contention_time = jiffies; -/* mb(); */ - obj->oo_contended =1 ; -} - -void osc_object_clear_contended (struct osc_object *obj) -{ - obj->oo_contended =0 ; -} -int osc_object_is_contended (struct osc_object *obj) {struct osc_device *dev =lu2osc_dev (obj->oo_cl.co_lu.lo_dev);
@@ -327,6 +328,7 @@ int osc_object_is_contended(struct osc_object *obj) }return 1 ; } +EXPORT_SYMBOL (osc_object_is_contended);/** * Implementation of struct cl_object_operations::coo_req_attr_set() for osc
@@ -438,6 +440,7 @@ struct lu_object *osc_object_alloc(const struct lu_env *env,lu_object_init (obj,NULL , dev); osc->oo_cl.co_ops = &osc_ops; obj->lo_ops = &osc_lu_obj_ops; + osc->oo_obj_ops = &osc_object_ops; }else { obj =NULL ; }
fs/lustre/osc/osc_request.c +6 -19
@@ -97,18 +97,6 @@ struct osc_ladvise_args {void *la_cookie; }; -struct osc_enqueue_args { -struct obd_export *oa_exp; -enum ldlm_type oa_type; -enum ldlm_mode oa_mode; -u64 *oa_flags; -osc_enqueue_upcall_f oa_upcall; -void *oa_cookie; -struct ost_lvb *oa_lvb; -struct lustre_handle oa_lockh; -unsigned int oa_speculative; -}; -static void osc_release_ppga (struct brw_page **ppga,u32 count);static int brw_interpret (const struct lu_env *env,struct ptlrpc_request *req,void *data,int rc);
@@ -2042,10 +2030,10 @@ static int osc_set_lock_data(struct ldlm_lock *lock, void *data)return set; } -static int osc_enqueue_fini (struct ptlrpc_request *req, -osc_enqueue_upcall_f upcall,void *cookie, -struct lustre_handle *lockh,enum ldlm_mode mode, -u64 *flags,int speculative,int errcode) +int osc_enqueue_fini (struct ptlrpc_request *req,osc_enqueue_upcall_f upcall, +void *cookie,struct lustre_handle *lockh, +enum ldlm_mode mode,u64 *flags,int speculative, +int errcode) {bool intent = *flags &LDLM_FL_HAS_INTENT ;int rc;
@@ -2077,9 +2065,8 @@ static int osc_enqueue_fini(struct ptlrpc_request *req,return rc; } -static int osc_enqueue_interpret (const struct lu_env *env, -struct ptlrpc_request *req, -struct osc_enqueue_args *aa,int rc) +int osc_enqueue_interpret (const struct lu_env *env,struct ptlrpc_request *req, +struct osc_enqueue_args *aa,int rc) {struct ldlm_lock *lock;struct lustre_handle *lockh = &aa->oa_lockh;
include/uapi/linux/lustre/lustre_idl.h +4 -0
@@ -1434,6 +1434,8 @@ enum mdt_reint_cmd {#define MDS_INODELOCK_MAXSHIFT 6/* This FULL lock is useful to take on unlink sort of operations */ #define MDS_INODELOCK_FULL ((1 << (MDS_INODELOCK_MAXSHIFT + 1)) - 1) +/* DOM lock shouldn't be canceled early, use this macro for ELC */ +#define MDS_INODELOCK_ELC (MDS_INODELOCK_FULL & ~MDS_INODELOCK_DOM)/* NOTE: until Lustre 1.8.7/2.1.1 the fid_ver() was packed into name[2], * but was moved into name[1] along with the OID to avoid consuming the
@@ -2153,6 +2155,8 @@ enum ldlm_intent_flags {IT_QUOTA_DQACQ =0x00000800 ,IT_QUOTA_CONN =0x00001000 ,IT_SETXATTR =0x00002000 , +IT_GLIMPSE =0x00004000 , + IT_BRW = 0x00008000, }; struct ldlm_intent {
Signature · 1 line
1.8.3.1
PATCH 49/151 lustre: mdc: add IO stats in mdc
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Add IO statistic into MDC like in OSC
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 2f103489f614 ("LU-3285 mdc: add IO stats in mdc")
Reviewed-on: https://review.whamcloud.com/28019
--- fs/lustre/include/lustre_dlm.h | 29 ++++ fs/lustre/include/lustre_osc.h | 2 + fs/lustre/ldlm/ldlm_request.c | 22 +++ fs/lustre/mdc/lproc_mdc.c | 318 ++++++++++++++++++++++++++++++++++++++++- fs/lustre/obdclass/genops.c | 2 + fs/lustre/osc/osc_internal.h | 2 - fs/lustre/osc/osc_page.c | 1 + 7 files changed, 367 insertions(+), 9 deletions(-)
fs/lustre/include/lustre_dlm.h +29 -0
@@ -1206,6 +1206,35 @@ void ldlm_namespace_free_prior(struct ldlm_namespace *ns,void ldlm_debugfs_setup (void );void ldlm_debugfs_cleanup (void ); +static inline void ldlm_svc_get_eopc (const struct ldlm_request *dlm_req, +struct lprocfs_stats *srv_stats) +{ +int lock_type =0 , op =0 ; + + lock_type = dlm_req->lock_desc.l_resource.lr_type; + +switch (lock_type) { +case LDLM_PLAIN : + op =PTLRPC_LAST_CNTR +LDLM_PLAIN_ENQUEUE ; +break ; +case LDLM_EXTENT : + op =PTLRPC_LAST_CNTR +LDLM_EXTENT_ENQUEUE ; +break ; +case LDLM_FLOCK : + op =PTLRPC_LAST_CNTR +LDLM_FLOCK_ENQUEUE ; +break ; +case LDLM_IBITS : + op =PTLRPC_LAST_CNTR +LDLM_IBITS_ENQUEUE ; +break ; +default : + op =0 ; +break ; + } + +if (op !=0 ) +lprocfs_counter_incr (srv_stats, op); +} +/* resource.c - internal */ struct ldlm_resource *ldlm_resource_get(struct ldlm_namespace *ns, struct ldlm_resource *parent,
fs/lustre/include/lustre_osc.h +2 -0
@@ -558,6 +558,8 @@ void osc_index2policy(union ldlm_policy_data *policy,void osc_page_submit (const struct lu_env *env,struct osc_page *opg,enum cl_req_type crt,int brw_flags);int lru_queue_work (const struct lu_env *env,void *data); +long osc_lru_shrink (const struct lu_env *env,struct client_obd *cli, +long target,bool force);/* osc_cache.c */ int osc_set_async_flags(struct osc_object *obj, struct osc_page *opg,
fs/lustre/ldlm/ldlm_request.c +22 -0
@@ -754,6 +754,28 @@ int ldlm_cli_enqueue(struct obd_export *exp, struct ptlrpc_request **reqp, body->lock_flags =ldlm_flags_to_wire (*flags); body->lock_handle[0 ] = *lockh; +/* extended LDLM opcodes in client stats */ +if (exp->exp_obd->obd_svc_stats !=NULL ) { +bool glimpse = *flags &LDLM_FL_HAS_INTENT ; + +/* OST glimpse has no intent buffer */ +if (req_capsule_has_field (&req->rq_pill, &RMF_LDLM_INTENT , +RCL_CLIENT )) { +struct ldlm_intent *it; + + it =req_capsule_client_get (&req->rq_pill, + &RMF_LDLM_INTENT ); + glimpse = (it && (it->opc ==IT_GLIMPSE )); + } + +if (!glimpse) +ldlm_svc_get_eopc (body, exp->exp_obd->obd_svc_stats); +else +lprocfs_counter_incr (exp->exp_obd->obd_svc_stats, +PTLRPC_LAST_CNTR + +LDLM_GLIMPSE_ENQUEUE ); + } +if (async) {LASSERT (reqp);return 0 ;
fs/lustre/mdc/lproc_mdc.c +311 -7
356 lines not on this page · raw message
fs/lustre/obdclass/genops.c +2 -0
@@ -1391,6 +1391,8 @@ int obd_set_max_rpcs_in_flight(struct client_obd *cli, u32 max)spin_lock (&cli->cl_loi_list_lock); old = cli->cl_max_rpcs_in_flight; cli->cl_max_rpcs_in_flight = max; +client_adjust_max_dirty (cli); + diff = max - old;/* We increase the max_rpcs_in_flight, then wakeup some waiters. */
fs/lustre/osc/osc_internal.h +0 -2
@@ -78,8 +78,6 @@ int osc_ladvise_base(struct obd_export *exp, struct obdo *oa,int osc_process_config_base (struct obd_device *obd,struct lustre_cfg *cfg);int osc_build_rpc (const struct lu_env *env,struct client_obd *cli,struct list_head *ext_list,int cmd); -long osc_lru_shrink (const struct lu_env *env,struct client_obd *cli, -long target,bool force);unsigned long osc_lru_reserve (struct client_obd *cli,unsigned long npages);void osc_lru_unreserve (struct client_obd *cli,unsigned long npages);
fs/lustre/osc/osc_page.c +1 -0
@@ -663,6 +663,7 @@ long osc_lru_shrink(const struct lu_env *env, struct client_obd *cli, }return count >0 ? count : rc; } +EXPORT_SYMBOL (osc_lru_shrink);/** * Reclaim LRU pages by an IO thread. The caller wants to reclaim at least
Signature · 1 line
1.8.3.1
PATCH 50/151 lustre: lov: add Data-on-MDT tests and fixes
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
- fixes for DoM code for issues revealed by tests.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: a7625cd2f37a ("LU-3285 test: add Data-on-MDT tests and fixes")
Reviewed-on: https://review.whamcloud.com/28020
--- fs/lustre/lov/lov_object.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
fs/lustre/lov/lov_object.c +17 -12
@@ -431,8 +431,17 @@ static int lov_attr_get_dom(const struct lu_env *env, struct lov_object *lov,return OST_LVB_GET_ERR (loi->loi_lvb.lvb_blocks);cl_lvb2attr (attr, &loi->loi_lvb); - attr->cat_kms = attr->cat_size > loi->loi_kms ? attr->cat_size : - loi->loi_kms; + +/* DoM component size can be bigger than stripe size after +* client's setattr RPC, so do not count anything beyound +* component end. Alternatively, check that limit on server +* and do not allow size overflow there. +*/ +if (attr->cat_size > lle->lle_extent.e_end) + attr->cat_size = lle->lle_extent.e_end; + + attr->cat_kms = attr->cat_size; + dom->lo_dom_r0.lo_attr_valid =1 ; *lov_attr = attr;
@@ -508,7 +517,6 @@ static int lov_init_dom(const struct lu_env *env, struct lov_device *dev,struct cl_device *mdcdev;struct lov_oinfo *loi =NULL ;struct cl_object_conf *sconf = <i->lti_stripe_conf; -struct inode *inode = conf->coc_inode;u32 idx =0 ;int rc;
@@ -538,15 +546,6 @@ static int lov_init_dom(const struct lu_env *env, struct lov_device *dev,return -ENOMEM ;fid_to_ostid (lu_object_fid (lov2lu (lov)), &loi->loi_oi); -/* Initialize lvb structure */ - loi->loi_lvb.lvb_mtime = inode->i_mtime.tv_sec; - loi->loi_lvb.lvb_atime = inode->i_atime.tv_sec; - loi->loi_lvb.lvb_ctime = inode->i_ctime.tv_sec; - loi->loi_lvb.lvb_blocks = inode->i_blocks; - loi->loi_lvb.lvb_size =i_size_read (inode); -if (loi->loi_lvb.lvb_size > lsme->lsme_stripe_size) - loi->loi_lvb.lvb_size = lsme->lsme_stripe_size; -loi_kms_set (loi, loi->loi_lvb.lvb_size); sconf->u.coc_oinfo = loi; again:
@@ -843,6 +842,12 @@ static int lov_attr_get_composite(const struct lu_env *env,if (!lov_attr)continue ; +CDEBUG (D_INODE , +"COMP ID #%i: s=%llu m=%llu a=%llu c=%llu b=%llu\n" , + index -1 , lov_attr->cat_size, + lov_attr->cat_mtime, lov_attr->cat_atime, + lov_attr->cat_ctime, lov_attr->cat_blocks); +/* merge results */ attr->cat_blocks += lov_attr->cat_blocks;if (attr->cat_size < lov_attr->cat_size)
Signature · 1 line
1.8.3.1
PATCH 51/151 lustre: mdc: use generic grant code at MDT
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Now that we have files on the MDT server we need to expand the mdc layer to handle grant space. Export osc_reconnect() and osc_disconnect() for the mdc layer. Indicate that we support grants on MDT with the client by adding OBD_CONNECT_GRANT_PARAM.
WC--bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 0697cf73c74d ("LU-3285 mdt: use generic grant code at MDT")
Reviewed-on: https://review.whamcloud.com/28021
--- fs/lustre/include/lustre_osc.h | 5 ++++- fs/lustre/llite/llite_lib.c | 3 ++- fs/lustre/mdc/mdc_request.c | 3 ++- fs/lustre/osc/osc_request.c | 12 ++++++------ 4 files changed, 14 insertions(+), 9 deletions(-)
fs/lustre/include/lustre_osc.h +4 -1
@@ -647,7 +647,10 @@ int osc_set_info_async(const struct lu_env *env, struct obd_export *exp, struct ptlrpc_request_set *set);int osc_ldlm_resource_invalidate (struct cfs_hash *hs,struct cfs_hash_bd *bd,struct hlist_node *hnode,void *arg); - +int osc_reconnect (const struct lu_env *env,struct obd_export *exp, +struct obd_device *obd,struct obd_uuid *cluuid, +struct obd_connect_data *data,void *localdata); +int osc_disconnect (struct obd_export *exp);int osc_punch_send (struct obd_export *exp,struct obdo *oa,obd_enqueue_update_f upcall,void *cookie);
fs/lustre/llite/llite_lib.c +2 -1
@@ -205,7 +205,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)OBD_CONNECT_DIR_STRIPE |OBD_CONNECT_BULK_MBITS |OBD_CONNECT_CKSUM |OBD_CONNECT_SUBTREE | -OBD_CONNECT_FLAGS2 |OBD_CONNECT_MULTIMODRPCS ; +OBD_CONNECT_MULTIMODRPCS | +OBD_CONNECT_GRANT_PARAM |OBD_CONNECT_FLAGS2 ; data->ocd_connect_flags2 =0 ;
fs/lustre/mdc/mdc_request.c +2 -1
@@ -2596,7 +2596,8 @@ int mdc_process_config(struct obd_device *obd, u32 len, void *buf) .add_conn = client_import_add_conn, .del_conn = client_import_del_conn, .connect = client_connect_import, - .disconnect = client_disconnect_export, + .reconnect = osc_reconnect, + .disconnect = osc_disconnect, .iocontrol = mdc_iocontrol, .set_info_async = mdc_set_info_async, .statfs = mdc_statfs,
fs/lustre/osc/osc_request.c +6 -6
@@ -2633,11 +2633,9 @@ int osc_set_info_async(const struct lu_env *env, struct obd_export *exp, }EXPORT_SYMBOL (osc_set_info_async); -static int osc_reconnect (const struct lu_env *env, -struct obd_export *exp,struct obd_device *obd, -struct obd_uuid *cluuid, -struct obd_connect_data *data, -void *localdata) +int osc_reconnect (const struct lu_env *env,struct obd_export *exp, +struct obd_device *obd,struct obd_uuid *cluuid, +struct obd_connect_data *data,void *localdata) {struct client_obd *cli = &obd->u.cli;
@@ -2663,8 +2661,9 @@ static int osc_reconnect(const struct lu_env *env,return 0 ; } +EXPORT_SYMBOL (osc_reconnect); -static int osc_disconnect (struct obd_export *exp) +int osc_disconnect (struct obd_export *exp) {struct obd_device *obd =class_exp2obd (exp);int rc;
@@ -2691,6 +2690,7 @@ static int osc_disconnect(struct obd_export *exp)osc_del_shrink_grant (&obd->u.cli);return rc; } +EXPORT_SYMBOL (osc_disconnect); int osc_ldlm_resource_invalidate(struct cfs_hash *hs, struct cfs_hash_bd *bd,
Signature · 1 line
1.8.3.1
PATCH 52/151 lustre: mds: combine DoM bit with other IBITS
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
The DoM bit combining with other ibits opens way to several optimization, like getting IO lock in advance along with OPEN, size attribute caching at client and others.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 238a309ea608 ("LU-3285 mds: combine DoM bit with other IBITS")
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Reviewed-on: https://review.whamcloud.com/28024
Lustre-commit: f8929e6d0f3c ("LU-3285 mdc: remove wrong assertion from mdc_object_ast_clear")
Reviewed-on: https://review.whamcloud.com/30021
--- fs/lustre/include/cl_object.h | 2 + fs/lustre/include/lustre_dlm.h | 2 + fs/lustre/ldlm/ldlm_inodebits.c | 22 ++++++++ fs/lustre/ldlm/ldlm_internal.h | 1 + fs/lustre/ldlm/ldlm_lock.c | 2 +- fs/lustre/llite/namei.c | 57 ++++++++++++++++++--- fs/lustre/lov/lov_object.c | 8 ++- fs/lustre/mdc/mdc_dev.c | 94 +++++++++++++++++++++++----------- fs/lustre/mdc/mdc_internal.h | 3 +- fs/lustre/mdc/mdc_locks.c | 50 +++++++++++++++--- fs/lustre/ptlrpc/pack_generic.c | 4 +- fs/lustre/ptlrpc/wiretest.c | 16 +++--- include/uapi/linux/lustre/lustre_idl.h | 7 +-- 13 files changed, 209 insertions(+), 59 deletions(-)
fs/lustre/include/cl_object.h +2 -0
@@ -286,6 +286,8 @@ struct cl_layout {struct lu_buf cl_buf;/** size of layout in lov_mds_md format. */ size_t cl_size; +/** size of DoM component if exists or zero otherwise */ +u32 cl_dom_comp_size;/** Layout generation. */ u32 cl_layout_gen;/** whether layout is a composite one */
fs/lustre/include/lustre_dlm.h +2 -0
@@ -1390,5 +1390,7 @@ static inline int ldlm_extent_contain(const struct ldlm_extent *ex1, return ex1->start <= ex2->start && ex1->end >= ex2->end; } +int ldlm_inodebits_drop (struct ldlm_lock *lock,__u64 to_drop); +#endif /** @} LDLM */
fs/lustre/ldlm/ldlm_inodebits.c +22 -0
@@ -67,3 +67,25 @@ void ldlm_ibits_policy_local_to_wire(const union ldlm_policy_data *lpolicy,memset (wpolicy,0 ,sizeof (*wpolicy)); wpolicy->l_inodebits.bits = lpolicy->l_inodebits.bits; } + +int ldlm_inodebits_drop (struct ldlm_lock *lock,__u64 to_drop) +{ +check_res_locked (lock->l_resource); + +/* Just return if there are no conflicting bits */ +if ((lock->l_policy_data.l_inodebits.bits & to_drop) ==0 ) { +LDLM_WARN (lock,"try to drop unset bits %#llx/%#llx\n" , + lock->l_policy_data.l_inodebits.bits, to_drop); +/* nothing to do */ +return 0 ; + } + +/* remove lock from a skiplist and put in the new place +* according with new inodebits +*/ +ldlm_resource_unlink_lock (lock); + lock->l_policy_data.l_inodebits.bits &= ~to_drop; +ldlm_grant_lock_with_skiplist (lock); +return 0 ; +} +EXPORT_SYMBOL (ldlm_inodebits_drop);
fs/lustre/ldlm/ldlm_internal.h +1 -0
@@ -128,6 +128,7 @@ enum ldlm_desc_ast_t { LDLM_WORK_GL_AST }; +void ldlm_grant_lock_with_skiplist (struct ldlm_lock *lock);void ldlm_grant_lock (struct ldlm_lock *lock,struct list_head *work_list);int ldlm_fill_lvb (struct ldlm_lock *lock,struct req_capsule *pill,enum req_location loc,void *data,int size);
fs/lustre/ldlm/ldlm_lock.c +1 -1
@@ -981,7 +981,7 @@ static void ldlm_granted_list_add_lock(struct ldlm_lock *lock,* Add a lock to granted list on a resource maintaining skiplist * correctness. */ -static void ldlm_grant_lock_with_skiplist (struct ldlm_lock *lock) +void ldlm_grant_lock_with_skiplist (struct ldlm_lock *lock) {struct sl_insert_point prev;
fs/lustre/llite/namei.c +51 -6
77 lines not on this page · raw message
fs/lustre/lov/lov_object.c +7 -1
@@ -1869,8 +1869,14 @@ static int lov_object_layout_get(const struct lu_env *env, cl->cl_size =lov_comp_md_size (lsm); cl->cl_layout_gen = lsm->lsm_layout_gen; - cl->cl_is_composite =lsm_is_composite (lsm->lsm_magic); +if (lsm_is_composite (lsm->lsm_magic)) { +struct lov_stripe_md_entry *lsme = lsm->lsm_entries[0 ]; + cl->cl_is_composite = true; + +if (lsme_is_dom (lsme)) + cl->cl_dom_comp_size = lsme->lsme_extent.e_end; + } rc =lov_lsm_pack (lsm, buf->lb_buf, buf->lb_len);lov_lsm_put (lsm);
fs/lustre/mdc/mdc_dev.c +64 -30
193 lines not on this page · raw message
fs/lustre/mdc/mdc_internal.h +2 -1
@@ -156,5 +156,6 @@ static inline unsigned long hash_x_index(u64 hash, int hash64)extern struct lu_device_type mdc_device_type;int mdc_ldlm_blocking_ast (struct ldlm_lock *dlmlock,struct ldlm_lock_desc *new,void *data,int flag); - +int mdc_ldlm_glimpse_ast (struct ldlm_lock *dlmlock,void *data); +int mdc_fill_lvb (struct ptlrpc_request *req,struct ost_lvb *lvb);#endif
fs/lustre/mdc/mdc_locks.c +43 -7
98 lines not on this page · raw message
fs/lustre/ptlrpc/pack_generic.c +2 -2
@@ -1758,8 +1758,8 @@ void lustre_swab_mdt_body(struct mdt_body *b)__swab32s (&b->mbo_uid_h);__swab32s (&b->mbo_gid_h);__swab32s (&b->mbo_projid); - BUILD_BUG_ON(offsetof(typeof(*b), mbo_padding_6) == 0); - BUILD_BUG_ON(offsetof(typeof(*b), mbo_padding_7) == 0); +__swab64s (&b->mbo_dom_size); +__swab64s (&b->mbo_dom_blocks); BUILD_BUG_ON(offsetof(typeof(*b), mbo_padding_8) == 0); BUILD_BUG_ON(offsetof(typeof(*b), mbo_padding_9) == 0); BUILD_BUG_ON(offsetof(typeof(*b), mbo_padding_10) == 0);
fs/lustre/ptlrpc/wiretest.c +8 -8
@@ -1982,14 +1982,14 @@ void lustre_assert_wire_constants(void) (long long)(int)offsetof(struct mdt_body, mbo_projid));LASSERTF ((int )sizeof (((struct mdt_body *)0 )->mbo_projid) ==4 ,"found %lld\n" , (long long )(int )sizeof (((struct mdt_body *)0 )->mbo_projid)); -LASSERTF ((int )offsetof(struct mdt_body , mbo_padding_6) ==176 ,"found %lld\n" , - (long long )(int )offsetof(struct mdt_body , mbo_padding_6)); -LASSERTF ((int )sizeof (((struct mdt_body *)0 )->mbo_padding_6) ==8 ,"found %lld\n" , - (long long )(int )sizeof (((struct mdt_body *)0 )->mbo_padding_6)); -LASSERTF ((int )offsetof(struct mdt_body , mbo_padding_7) ==184 ,"found %lld\n" , - (long long )(int )offsetof(struct mdt_body , mbo_padding_7)); -LASSERTF ((int )sizeof (((struct mdt_body *)0 )->mbo_padding_7) ==8 ,"found %lld\n" , - (long long )(int )sizeof (((struct mdt_body *)0 )->mbo_padding_7)); +LASSERTF ((int )offsetof(struct mdt_body , mbo_dom_size) ==176 ,"found %lld\n" , + (long long )(int )offsetof(struct mdt_body , mbo_dom_size)); +LASSERTF ((int )sizeof (((struct mdt_body *)0 )->mbo_dom_size) ==8 ,"found %lld\n" , + (long long )(int )sizeof (((struct mdt_body *)0 )->mbo_dom_size)); +LASSERTF ((int )offsetof(struct mdt_body , mbo_dom_blocks) ==184 ,"found %lld\n" , + (long long )(int )offsetof(struct mdt_body , mbo_dom_blocks)); +LASSERTF ((int )sizeof (((struct mdt_body *)0 )->mbo_dom_blocks) ==8 ,"found %lld\n" , + (long long )(int )sizeof (((struct mdt_body *)0 )->mbo_dom_blocks));LASSERTF ((int )offsetof(struct mdt_body , mbo_padding_8) ==192 ,"found %lld\n" , (long long )(int )offsetof(struct mdt_body , mbo_padding_8)); LASSERTF((int)sizeof(((struct mdt_body *)0)->mbo_padding_8) == 8, "found %lld\n",
include/uapi/linux/lustre/lustre_idl.h +4 -3
@@ -1088,6 +1088,7 @@ static inline __u32 lov_mds_md_size(__u16 stripes, __u32 lmm_magic)#define OBD_MD_FLUID (0x00000200ULL)/* user ID */ #define OBD_MD_FLGID (0x00000400ULL)/* group ID */ #define OBD_MD_FLFLAGS (0x00000800ULL)/* flags word */ +#define OBD_MD_DOM_SIZE (0x00001000ULL)/* Data-on-MDT component size */ #define OBD_MD_FLNLINK (0x00002000ULL)/* link count */ #define OBD_MD_FLGENER (0x00004000ULL)/* generation number */ /*#define OBD_MD_FLINLINE (0x00008000ULL) inline data. used until 1.6.5 */
@@ -1547,9 +1548,9 @@ struct mdt_body {__u32 mbo_uid_h;/* high 32-bits of uid, for FUID */ __u32 mbo_gid_h;/* high 32-bits of gid, for FUID */ __u32 mbo_projid;/* also fix lustre_swab_mdt_body */ -__u64 mbo_padding_6; -__u64 mbo_padding_7; -__u64 mbo_padding_8; +__u64 mbo_dom_size;/* size of DOM component */ +__u64 mbo_dom_blocks;/* blocks consumed by DOM component */ +__u64 mbo_padding_8;/* also fix lustre_swab_mdt_body */ __u64 mbo_padding_9;__u64 mbo_padding_10; };/* 216 */
Signature · 1 line
1.8.3.1
PATCH 53/151 lustre: llite: increase whole-file readahead to RPC size
In reply to James SimmonsFrom: Andreas Dilger <[email protected]>
Increase the default whole-file readahead limit to match the current RPC size. That ensures that files smaller than the RPC size will be read in a single round-trip instead of sending multiple smaller RPCs.
WC-bug-id: https://jira.whamcloud.com/browse/LU-7990
Lustre-commit: 627d0133d9d7 ("LU-7990 llite: increase whole-file readahead to RPC size")
Reviewed-on: https://review.whamcloud.com/26955
--- fs/lustre/llite/llite_lib.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
fs/lustre/llite/llite_lib.c +7 -2
@@ -95,8 +95,7 @@ static struct ll_sb_info *ll_init_sbi(void) sbi->ll_ra_info.ra_max_pages_per_file =min (pages /32 ,SBI_DEFAULT_READAHEAD_MAX ); sbi->ll_ra_info.ra_max_pages = sbi->ll_ra_info.ra_max_pages_per_file; - sbi->ll_ra_info.ra_max_read_ahead_whole_pages = -SBI_DEFAULT_READAHEAD_WHOLE_MAX ; + sbi->ll_ra_info.ra_max_read_ahead_whole_pages =-1 ;ll_generate_random_uuid (uuid);sprintf (sbi->ll_sb_uuid.uuid,"%pU" , uuid);
@@ -269,6 +268,12 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt) sbi->ll_md_exp->exp_connect_data = *data; +/* Don't change value if it was specified in the config log */ +if (sbi->ll_ra_info.ra_max_read_ahead_whole_pages ==-1 ) + sbi->ll_ra_info.ra_max_read_ahead_whole_pages = + max_t(unsigned long, SBI_DEFAULT_READAHEAD_WHOLE_MAX, + (data->ocd_brw_size >>PAGE_SHIFT )); + err =obd_fid_init (sbi->ll_md_exp->exp_obd, sbi->ll_md_exp,LUSTRE_SEQ_METADATA );if (err) {
Signature · 1 line
1.8.3.1
PATCH 54/151 lustre: ldlm: remove liblustre remnants
In reply to James SimmonsFrom: Andreas Dilger <[email protected]>
Remove the last remnants of liblustre in the code. There are still some liblustre remnants in the code (e.g. blocked lock handling for LDLM_FL_CANCEL_ON_BLOCK), but that has more complex semantics and should be removed in a separate patch.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9452
Lustre-commit: 82e794e268be ("LU-9452 ldlm: remove MSG_CONNECT_LIBCLIENT support")
Reviewed-on: https://review.whamcloud.com/26972
--- fs/lustre/include/lustre_dlm_flags.h | 4 ++-- fs/lustre/llite/glimpse.c | 3 +-- fs/lustre/llite/lcommon_cl.c | 3 +-- fs/lustre/llite/lcommon_misc.c | 3 +-- fs/lustre/obdclass/cl_object.c | 12 +++++------- fs/lustre/ptlrpc/import.c | 12 ++++++------ fs/lustre/ptlrpc/niobuf.c | 2 +- 7 files changed, 17 insertions(+), 22 deletions(-)
fs/lustre/include/lustre_dlm_flags.h +2 -2
@@ -137,8 +137,8 @@/** * Immediately cancel such locks when they block some other locks. Send * cancel notification to original lock holder, but expect no reply. This -* is for clients (like liblustre) that cannot be expected to reliably -* response to blocking AST. +* is for clients that cannot be expected to reliably response to blocking +* AST. */ #define LDLM_FL_CANCEL_ON_BLOCK 0x0000000000800000ULL/* bit 23 */ #define ldlm_is_cancel_on_block (_l) LDLM_TEST_FLAG((_l), 1ULL << 23)
fs/lustre/llite/glimpse.c +1 -2
@@ -30,8 +30,7 @@* This file is part of Lustre, http://www.lustre.org/ * Lustre is a trademark of Sun Microsystems, Inc. * -* glimpse code shared between vvp and liblustre (and other Lustre clients in -* the future). +* glimpse code used by vvp (and other Lustre clients in the future). * * Author: Nikita Danilov <[email protected]> * Author: Oleg Drokin <[email protected]>
fs/lustre/llite/lcommon_cl.c +1 -2
@@ -30,8 +30,7 @@* This file is part of Lustre, http://www.lustre.org/ * Lustre is a trademark of Sun Microsystems, Inc. * -* cl code shared between vvp and liblustre (and other Lustre clients in the -* future). +* cl code used by vvp (and other Lustre clients in the future). * * Author: Nikita Danilov <[email protected]> */
fs/lustre/llite/lcommon_misc.c +1 -2
@@ -30,8 +30,7 @@* This file is part of Lustre, http://www.lustre.org/ * Lustre is a trademark of Sun Microsystems, Inc. * -* cl code shared between vvp and liblustre (and other Lustre clients in the -* future). +* cl code used by vvp (and other Lustre clients in the future). * */ #define DEBUG_SUBSYSTEM S_LLITE
fs/lustre/obdclass/cl_object.c +5 -7
@@ -515,18 +515,16 @@ int cl_site_stats_print(const struct cl_site *site, struct seq_file *m)/** * The most efficient way is to store cl_env pointer in task specific -* structures. On Linux, it wont' be easy to use task_struct->journal_info -* because Lustre code may call into other fs which has certain assumptions -* about journal_info. Currently following fields in task_struct are identified -* can be used for this purpose: -* - tux_info: only on RedHat kernel. -* - ... +* structures. On Linux, it isn't easy to use task_struct->journal_info +* because Lustre code may call into other fs during memory reclaim, which +* has certain assumptions about journal_info. There are not currently any +* fields in task_struct that can be used for this purpose. * \note As long as we use task_struct to store cl_env, we assume that once * called into Lustre, we'll never call into the other part of the kernel * which will use those fields in task_struct without explicitly exiting * Lustre. * -* If there's no space in task_struct is available, hash will be used. +* Since there's no space in task_struct is available, hash will be used. * bz20044, bz22683. */
fs/lustre/ptlrpc/import.c +6 -6
@@ -781,7 +781,7 @@ static int ptlrpc_connect_set_flags(struct obd_import *imp,* of macro arguments */ const char *older ="older than client. Consider upgrading server" ; -const char *newer ="newer than client. Consider recompiling application" ; +const char *newer ="newer than client. Consider upgrading client" ; LCONSOLE_WARN("Server %s version (%d.%d.%d.%d) is much %s (%s)\n", obd2cli_tgt(imp->imp_obd),
@@ -1237,14 +1237,14 @@ static int ptlrpc_connect_interpret(const struct lu_env *env, ocd =req_capsule_server_get (&request->rq_pill, &RMF_CONNECT_DATA ); +/* Servers are not supposed to refuse connections from +* clients based on version, only connection feature +* flags. We should never see this from llite, but it +* may be useful for debugging in the future. +*/ if (ocd && (ocd->ocd_connect_flags &OBD_CONNECT_VERSION ) && (ocd->ocd_version !=LUSTRE_VERSION_CODE )) { -/* -* Actually servers are only supposed to refuse -* connection from liblustre clients, so we -* should never see this from VFS context -*/ LCONSOLE_ERROR_MSG(0x16a, "Server %s version (%d.%d.%d.%d) refused connection from this client with an incompatible version (%s). Client must be recompiled\n", obd2cli_tgt(imp->imp_obd),
fs/lustre/ptlrpc/niobuf.c +1 -1
@@ -256,7 +256,7 @@ int ptlrpc_unregister_bulk(struct ptlrpc_request *req, int async)/* the unlink ensures the callback happens ASAP and is the last * one. If it fails, it must be because completion just happened, -* but we must still wait_event() in this case to give liblustre +* but we must still wait_event() in this case to give lustre * a chance to run client_bulk_callback() */ mdunlink_iterate_helper (desc->bd_mds, desc->bd_md_max_brw);
Signature · 1 line
1.8.3.1
PATCH 55/151 lustre: misc: replace LASSERT() with BUILD_BUG_ON()
In reply to James SimmonsFrom: Andreas Dilger <[email protected]>
Some code consistency checks are being done at runtime with LASSERT() when they could be done at compile time with BUILD_BUG_ON(). This might miss defects introduced into the code if that particular code path is not exercised during testing.
Replace LASSERT() with BUILD_BUG_ON() in such cases.
WC-bug-id: https://jira.whamcloud.com/browse/LU-10046
Lustre-commit: 558c93dc56dc ("LU-10046 misc: replace LASSERT() with CLASSERT()")
Reviewed-on: https://review.whamcloud.com/29256
--- fs/lustre/llite/dir.c | 6 +++--- fs/lustre/ptlrpc/client.c | 2 +- fs/lustre/ptlrpc/lproc_ptlrpc.c | 1 + 3 files changed, 5 insertions(+), 4 deletions(-)
fs/lustre/llite/dir.c +3 -3
@@ -1229,9 +1229,9 @@ static long ll_dir_ioctl(struct file *file, unsigned int cmd, unsigned long arg)BUILD_BUG_ON (sizeof (struct lov_user_md_v3 ) <=sizeof (struct lov_comp_md_v1 )); -LASSERT (sizeof (lumv3) ==sizeof (*lumv3p)); -LASSERT (sizeof (lumv3.lmm_objects[0 ]) == -sizeof (lumv3p->lmm_objects[0 ])); +BUILD_BUG_ON (sizeof (lumv3) !=sizeof (*lumv3p)); +BUILD_BUG_ON (sizeof (lumv3.lmm_objects[0 ]) != +sizeof (lumv3p->lmm_objects[0 ]));/* first try with v1 which is smaller than v3 */ if (copy_from_user (lumv1, lumv1p,sizeof (*lumv1)))return -EFAULT ;
fs/lustre/ptlrpc/client.c +1 -1
@@ -2934,7 +2934,7 @@ int ptlrpc_replay_req(struct ptlrpc_request *req)LASSERT (req->rq_import->imp_state ==LUSTRE_IMP_REPLAY ); -LASSERT (sizeof (*aa) <=sizeof (req->rq_async_args)); +BUILD_BUG_ON (sizeof (*aa) >sizeof (req->rq_async_args)); aa =ptlrpc_req_async_args (req);memset (aa,0 ,sizeof (*aa));
fs/lustre/ptlrpc/lproc_ptlrpc.c +1 -0
@@ -449,6 +449,7 @@ static void nrs_policy_get_info_locked(struct ptlrpc_nrs_policy *policy, {assert_spin_locked (&policy->pol_nrs->nrs_lock); +BUILD_BUG_ON (sizeof (info->pi_arg) !=sizeof (policy->pol_arg));memcpy (info->pi_name, policy->pol_desc->pd_name,NRS_POL_NAME_MAX ); info->pi_fallback = !!(policy->pol_flags &PTLRPC_NRS_FL_FALLBACK );
Signature · 1 line
1.8.3.1
PATCH 56/151 lustre: llite: check layout size after cl_object_layout_get
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
Check that layout size is not zero before doing flush for DoM object. Also patch does that:
- initializes all values in cl_layout explicitly in lov_object_layout_get().
- changes cl_dom_comp_size to u64 for future needs, e.g. importing an existing filesystem image directly.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 877669a282ef ("LU-3285 llite: check layout size after cl_object_layout_get")
Reviewed-on: https://review.whamcloud.com/29810
--- fs/lustre/include/cl_object.h | 2 +- fs/lustre/llite/namei.c | 3 +-- fs/lustre/lov/lov_object.c | 4 ++++ 3 files changed, 6 insertions(+), 3 deletions(-)
fs/lustre/include/cl_object.h +1 -1
@@ -287,7 +287,7 @@ struct cl_layout {/** size of layout in lov_mds_md format. */ size_t cl_size;/** size of DoM component if exists or zero otherwise */ -u32 cl_dom_comp_size; +u64 cl_dom_comp_size;/** Layout generation. */ u32 cl_layout_gen;/** whether layout is a composite one */
fs/lustre/llite/namei.c +1 -2
@@ -197,10 +197,9 @@ int ll_dom_lock_cancel(struct inode *inode, struct ldlm_lock *lock)CDEBUG (D_INODE ,"Cannot get layout for " DFID "\n" ,PFID (ll_inode2fid (inode))); rc = -ENODATA ; - }else if (clt.cl_dom_comp_size ==0 ) { + }else if (clt.cl_size ==0 || clt.cl_dom_comp_size ==0 ) {CDEBUG (D_INODE ,"DOM lock without DOM layout for " DFID "\n" ,PFID (ll_inode2fid (inode))); - rc = -EINVAL ; }else {enum cl_fsync_mode mode;loff_t end = clt.cl_dom_comp_size -1 ;
fs/lustre/lov/lov_object.c +4 -0
@@ -1869,6 +1869,7 @@ static int lov_object_layout_get(const struct lu_env *env, cl->cl_size =lov_comp_md_size (lsm); cl->cl_layout_gen = lsm->lsm_layout_gen; + cl->cl_dom_comp_size =0 ;if (lsm_is_composite (lsm->lsm_magic)) {struct lov_stripe_md_entry *lsme = lsm->lsm_entries[0 ];
@@ -1876,7 +1877,10 @@ static int lov_object_layout_get(const struct lu_env *env,if (lsme_is_dom (lsme)) cl->cl_dom_comp_size = lsme->lsme_extent.e_end; + }else { + cl->cl_is_composite = false; } + rc =lov_lsm_pack (lsm, buf->lb_buf, buf->lb_len);lov_lsm_put (lsm);
Signature · 1 line
1.8.3.1
PATCH 57/151 lustre: mdc: implement own mdc_io_fsync_start()
In reply to James SimmonsFrom: Mikhal Pershin <[email protected]>
DoM lock cancellation may happen in llite and call cl_sync_file_range() function to flush related data and it uses DOM component end as limit for data to be flushed. However related lock and extent are expanded to EOF and this is asserted in osc_cache_writeback_range().
To avoid this a MDC uses own version of cio_start for FSYNC and osc_cache_writeback_range() is called on whole DoM object no matter what start/end are supplied by upper layers.
WC-bug-id: https://jira.whamcloud.com/browse/LU-3285
Lustre-commit: 1d2e9f42027c ("LU-3285 mdc: implement own mdc_io_fsync_start()")
Reviewed-on: https://review.whamcloud.com/29813
--- fs/lustre/include/lustre_osc.h | 4 ++-- fs/lustre/mdc/mdc_dev.c | 38 +++++++++++++++++++++++++++++++++++++- fs/lustre/osc/osc_io.c | 6 +++--- 3 files changed, 42 insertions(+), 6 deletions(-)
fs/lustre/include/lustre_osc.h +2 -2
@@ -676,8 +676,8 @@ int osc_io_read_start(const struct lu_env *env,int osc_io_write_start (const struct lu_env *env,const struct cl_io_slice *slice);void osc_io_end (const struct lu_env *env,const struct cl_io_slice *slice); -int osc_io_fsync_start (const struct lu_env *env, -const struct cl_io_slice *slice); +int osc_fsync_ost (const struct lu_env *env,struct osc_object *obj, +struct cl_fsync_io *fio);void osc_io_fsync_end (const struct lu_env *env,const struct cl_io_slice *slice);void osc_read_ahead_release (const struct lu_env *env,void *cbdata);
fs/lustre/mdc/mdc_dev.c +37 -1
@@ -1045,6 +1045,42 @@ static int mdc_io_read_ahead(const struct lu_env *env,return 0 ; } +int mdc_io_fsync_start (const struct lu_env *env, +const struct cl_io_slice *slice) +{ +struct cl_io *io = slice->cis_io; +struct cl_fsync_io *fio = &io->u.ci_fsync; +struct cl_object *obj = slice->cis_obj; +struct osc_object *osc =cl2osc (obj); +int result =0 ; + +/* a MDC lock always covers whole object, do sync for whole +* possible range despite of supplied start/end values. +*/ + result =osc_cache_writeback_range (env, osc,0 ,CL_PAGE_EOF ,0 , + fio->fi_mode ==CL_FSYNC_DISCARD ); +if (result >0 ) { + fio->fi_nr_written += result; + result =0 ; + } +if (fio->fi_mode ==CL_FSYNC_ALL ) { +int rc; + + rc =osc_cache_wait_range (env, osc,0 ,CL_PAGE_EOF ); +if (result ==0 ) + result = rc; +/* Use OSC sync code because it is asynchronous. +* It is to be added into MDC and avoid the using of +* OST_SYNC at both MDC and MDT. +*/ + rc =osc_fsync_ost (env, osc, fio); +if (result ==0 ) + result = rc; + } + +return result; +} +static struct cl_io_operations mdc_io_ops = { .op = { [CIT_READ ] = {
@@ -1076,7 +1112,7 @@ static int mdc_io_read_ahead(const struct lu_env *env, .cio_end = osc_io_end, }, [CIT_FSYNC] = { - .cio_start = osc_io_fsync_start, + .cio_start = mdc_io_fsync_start, .cio_end = osc_io_fsync_end, }, },
fs/lustre/osc/osc_io.c +3 -3
@@ -750,8 +750,8 @@ int osc_io_write_start(const struct lu_env *env, }EXPORT_SYMBOL (osc_io_write_start); -static int osc_fsync_ost (const struct lu_env *env,struct osc_object *obj, -struct cl_fsync_io *fio) +int osc_fsync_ost (const struct lu_env *env,struct osc_object *obj, +struct cl_fsync_io *fio) {struct osc_io *oio =osc_env_io (env);struct obdo *oa = &oio->oi_oa;
@@ -775,6 +775,7 @@ static int osc_fsync_ost(const struct lu_env *env, struct osc_object *obj, rc =osc_sync_base (obj, oa, osc_async_upcall, cbargs,PTLRPCD_SET );return rc; } +EXPORT_SYMBOL (osc_fsync_ost);int osc_io_fsync_start (const struct lu_env *env,const struct cl_io_slice *slice)
@@ -815,7 +816,6 @@ int osc_io_fsync_start(const struct lu_env *env,return result; } -EXPORT_SYMBOL (osc_io_fsync_start);void osc_io_fsync_end (const struct lu_env *env,const struct cl_io_slice *slice)
Signature · 1 line
1.8.3.1
PATCH 58/151 lustre: ldlm: migrate the rest of the code to 64 bit time
In reply to James SimmonsReplace the last cfs_time_current_sec() to avoid the overflow issues in 2038 with ktime_get_real_seconds(). Reduce the jiffies usage to the bare minimum which is useage for mod_timer() and schedule_timeout(). This makes the ldlm totally 64 bit time compliant.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9019
Lustre-commit: e920be681451 ("LU-9019 ldlm: migrate the rest of the code to 64 bit time")
Reviewed-on: https://review.whamcloud.com/29295
--- fs/lustre/include/lustre_dlm.h | 4 ++-- fs/lustre/include/lustre_net.h | 11 +++++------ fs/lustre/ldlm/ldlm_lock.c | 10 +++++----- fs/lustre/ldlm/ldlm_lockd.c | 2 +- fs/lustre/ldlm/ldlm_request.c | 14 +++++++------- fs/lustre/ldlm/ldlm_resource.c | 8 +++----- fs/lustre/ptlrpc/client.c | 4 ++-- fs/lustre/ptlrpc/import.c | 2 +- 8 files changed, 26 insertions(+), 29 deletions(-)
fs/lustre/include/lustre_dlm.h +2 -2
@@ -418,7 +418,7 @@ struct ldlm_namespace {* \see ldlm_namespace_dump. Increased by 10 seconds every time * it is called. */ -unsigned long ns_next_dump; +time64_t ns_next_dump;/** * LVB operations for this namespace.
@@ -754,7 +754,7 @@ struct ldlm_lock {* under this lock. * \see ost_rw_prolong_locks */ -unsigned long l_callback_timeout; +time64_t l_callback_timeout;/** Local PID of process which created this lock. */ u32 l_pid;
fs/lustre/include/lustre_net.h +5 -6
@@ -534,9 +534,9 @@ struct ptlrpc_cli_req {/** For bulk requests on client only: bulk descriptor */ struct ptlrpc_bulk_desc *cr_bulk;/** optional time limit for send attempts */ -long cr_delay_limit; +time64_t cr_delay_limit;/** time request was first queued */ -unsigned long cr_queued_time; +time64_t cr_queued_time;/** request sent in nanosseconds */ ktime_t cr_sent_ns;/** time for request really sent out */
@@ -867,8 +867,9 @@ struct ptlrpc_request {/** * service time estimate (secs) * If the request is not served by this time, it is marked as timed out. +* Do not change to time64_t since this is transmitted over the wire. */ -int rq_timeout; +time_t rq_timeout;/** * when request/reply sent (secs), or time when request should be sent */
@@ -2221,10 +2222,8 @@ static inline int ptlrpc_req_get_repsize(struct ptlrpc_request *req)static inline int ptlrpc_send_limit_expired (struct ptlrpc_request *req) {if (req->rq_delay_limit !=0 && -time_before (req->rq_queued_time + req->rq_delay_limit *HZ , - jiffies)) { + req->rq_queued_time + req->rq_delay_limit <ktime_get_seconds ())return 1 ; - }return 0 ; }
fs/lustre/ldlm/ldlm_lock.c +5 -5
@@ -1996,7 +1996,7 @@ void _ldlm_lock_debug(struct ldlm_lock *lock,if (!resource) { libcfs_debug_msg(msgdata, - "%pV ns: \?\? lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: \?\? rrc=\?\? type: \?\?\? flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lu lvb_type: %d\n", + "%pV ns: \?\? lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: \?\? rrc=\?\? type: \?\?\? flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lld lvb_type: %d\n", &vaf, lock, lock->l_handle.h_cookie,
@@ -2016,7 +2016,7 @@ void _ldlm_lock_debug(struct ldlm_lock *lock,switch (resource->lr_type) {case LDLM_EXTENT : libcfs_debug_msg(msgdata, - "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " rrc: %d type: %s [%llu->%llu] (req %llu->%llu) flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lu lvb_type: %d\n", + "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " rrc: %d type: %s [%llu->%llu] (req %llu->%llu) flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lld lvb_type: %d\n", &vaf, ldlm_lock_to_ns_name(lock), lock, lock->l_handle.h_cookie,
@@ -2040,7 +2040,7 @@ void _ldlm_lock_debug(struct ldlm_lock *lock,case LDLM_FLOCK : libcfs_debug_msg(msgdata, - "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " rrc: %d type: %s pid: %d [%llu->%llu] flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lu\n", + "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " rrc: %d type: %s pid: %d [%llu->%llu] flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lld\n", &vaf, ldlm_lock_to_ns_name(lock), lock, lock->l_handle.h_cookie,
@@ -2062,7 +2062,7 @@ void _ldlm_lock_debug(struct ldlm_lock *lock,case LDLM_IBITS : libcfs_debug_msg(msgdata, - "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " bits %#llx rrc: %d type: %s flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lu lvb_type: %d\n", + "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " bits %#llx rrc: %d type: %s flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lld lvb_type: %d\n", &vaf, ldlm_lock_to_ns_name(lock), lock, lock->l_handle.h_cookie,
@@ -2083,7 +2083,7 @@ void _ldlm_lock_debug(struct ldlm_lock *lock,default : libcfs_debug_msg(msgdata, - "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " rrc: %d type: %s flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lu lvb_type: %d\n", + "%pV ns: %s lock: %p/%#llx lrc: %d/%d,%d mode: %s/%s res: " DLDLMRES " rrc: %d type: %s flags: %#llx nid: %s remote: %#llx expref: %d pid: %u timeout: %lld lvb_type: %d\n", &vaf, ldlm_lock_to_ns_name(lock), lock, lock->l_handle.h_cookie,
fs/lustre/ldlm/ldlm_lockd.c +1 -1
@@ -160,7 +160,7 @@ static void ldlm_handle_cp_callback(struct ptlrpc_request *req,LDLM_DEBUG (lock,"client completion callback handler START" );if (OBD_FAIL_CHECK (OBD_FAIL_LDLM_CANCEL_BL_CB_RACE )) { -int to =HZ ; +long to =HZ ;while (to >0 ) {schedule_timeout_interruptible (to);
fs/lustre/ldlm/ldlm_request.c +7 -7
@@ -110,16 +110,16 @@ static void ldlm_expired_completion_wait(struct ldlm_lock *lock, u32 conn_cnt)struct obd_device *obd;if (!lock->l_conn_export) { -static unsigned long next_dump, last_dump; +static time64_t next_dump, last_dump;LDLM_ERROR (lock,"lock timed out (enqueued at %lld, %llds ago); not entering recovery in server code, just going back to sleep" , (s64 )lock->l_last_activity, (s64)(ktime_get_real_seconds () - lock->l_last_activity)); -if (time_after (jiffies, next_dump)) { +if (ktime_get_seconds () > next_dump) { last_dump = next_dump; - next_dump = jiffies +300 *HZ ; + next_dump =ktime_get_seconds () +300 ;ldlm_namespace_dump (D_DLMTRACE ,ldlm_lock_to_ns (lock));if (last_dump ==0 )
@@ -150,9 +150,9 @@ static void ldlm_expired_completion_wait(struct ldlm_lock *lock, u32 conn_cnt)/* We use the same basis for both server side and client side functions * from a single node. */ -static unsigned int ldlm_cp_timeout (struct ldlm_lock *lock) +static time64_t ldlm_cp_timeout (struct ldlm_lock *lock) { -unsigned int timeout; +time64_t timeout;if (AT_OFF )return obd_timeout;
@@ -163,7 +163,7 @@ static unsigned int ldlm_cp_timeout(struct ldlm_lock *lock)* doesn't respond reasonably, and then give us the lock. */ timeout =at_get (ldlm_lock_to_ns_at (lock)); -return max (3 * timeout, ldlm_enqueue_min); +return max (3 * timeout, (time64_t ) ldlm_enqueue_min); }/**
@@ -218,7 +218,7 @@ int ldlm_completion_ast(struct ldlm_lock *lock, u64 flags, void *data)/* XXX ALLOCATE - 160 bytes */ struct obd_device *obd;struct obd_import *imp =NULL ; -u32 timeout; +time64_t timeout;u32 conn_cnt =0 ;int rc =0 ;
fs/lustre/ldlm/ldlm_resource.c +3 -5
@@ -274,7 +274,6 @@ static ssize_t lru_max_age_store(struct kobject *kobj, struct attribute *attr,int scale =NSEC_PER_MSEC ;unsigned long long tmp;char *buf; -int err;/* Did the user ask in seconds or milliseconds. Default is in ms */ buf =strstr (buffer,"ms" );
@@ -287,8 +286,7 @@ static ssize_t lru_max_age_store(struct kobject *kobj, struct attribute *attr,if (buf) *buf ='\0' ; - err =kstrtoull (buffer,10 , &tmp); -if (err !=0 ) +if (kstrtoull (buffer,10 , &tmp))return -EINVAL ; ns->ns_max_age =ktime_set (0 , tmp * scale);
@@ -1283,14 +1281,14 @@ void ldlm_namespace_dump(int level, struct ldlm_namespace *ns)CDEBUG (level,"--- Namespace: %s (rc: %d, side: client)\n" ,ldlm_ns_name (ns),atomic_read (&ns->ns_bref)); -if (time_before (jiffies, ns->ns_next_dump)) +if (ktime_get_seconds () < ns->ns_next_dump)return ;cfs_hash_for_each_nolock (ns->ns_rs_hash, ldlm_res_hash_dump, (void *)(unsigned long )level,0 );spin_lock (&ns->ns_lock); - ns->ns_next_dump = jiffies +10 *HZ ; + ns->ns_next_dump =ktime_get_seconds () +10 ;spin_unlock (&ns->ns_lock); }
fs/lustre/ptlrpc/client.c +2 -2
@@ -1084,7 +1084,7 @@ void ptlrpc_set_add_req(struct ptlrpc_request_set *set,list_add_tail (&req->rq_set_chain, &set->set_requests); req->rq_set = set;atomic_inc (&set->set_remaining); - req->rq_queued_time = jiffies; + req->rq_queued_time =ktime_get_seconds ();if (req->rq_reqmsg)lustre_msg_set_jobid (req->rq_reqmsg,NULL );
@@ -1115,7 +1115,7 @@ void ptlrpc_set_add_new_req(struct ptlrpcd_ctl *pc,spin_lock (&set->set_new_req_lock);/* The set takes over the caller's request reference. */ req->rq_set = set; - req->rq_queued_time = jiffies; + req->rq_queued_time =ktime_get_seconds ();list_add_tail (&req->rq_set_chain, &set->set_new_requests); count =atomic_inc_return (&set->set_new_count);spin_unlock (&set->set_new_req_lock);
fs/lustre/ptlrpc/import.c +1 -1
@@ -717,7 +717,7 @@ int ptlrpc_connect_import(struct obd_import *imp)lustre_msg_add_op_flags (request->rq_reqmsg,MSG_CONNECT_TRANSNO ); -DEBUG_REQ (D_RPCTRACE , request,"(re)connect request (timeout %d)" , +DEBUG_REQ (D_RPCTRACE , request,"(re)connect request (timeout %ld)" , request->rq_timeout);ptlrpcd_add_req (request); rc =0 ;
Signature · 1 line
1.8.3.1
PATCH 59/151 lustre: llite: sync bdi sysfs name with lustre sysfs tree
In reply to James SimmonsWhen the linux kernel introduced super_setup_bdi_name() the port for the lustre client was done which ends up creating sysfs files of the format
/sys/devices/virtual/bdi/lustre-1
but their is not way to correlate this with the naming of lustre sysfs objects. Update the bdi handle to use the naming method applied to the llite sysfs tree.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9558
Lustre-commit: 1defa62a19e7 ("LU-9558 llite: port lustre to unified handling of bdi")
Reviewed-on: https://review.whamcloud.com/28511
--- fs/lustre/llite/llite_lib.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
fs/lustre/llite/llite_lib.c +1 -3
@@ -945,7 +945,6 @@ int ll_fill_super(struct super_block *sb)char *ptr;int len;int err; -static atomic_t ll_bdi_num =ATOMIC_INIT (0 );CDEBUG (D_VFSTRACE ,"VFS Op: sb %p\n" , sb);
@@ -970,8 +969,7 @@ int ll_fill_super(struct super_block *sb)if (err) goto out_free; - err =super_setup_bdi_name (sb,"lustre-%d" , -atomic_inc_return (&ll_bdi_num)); + err =super_setup_bdi_name (sb,"lustre-%p" , sb);if (err) goto out_free;
Signature · 1 line
1.8.3.1
PATCH 60/151 lustre: lov: allow lov.*.stripe{size, count}=-1 param
In reply to James SimmonsFrom: Andreas Dilger <[email protected]>
Both the lov stripecount and stripeoffset sysfs files allows the value of "-1". Setting it to "-1" to indicate that the filesystem-wide default value should be used. Update the ld_default_* setting to the proper signed type.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9611
Lustre-commit: f93276d9a5b5 ("LU-9611 lov: allow lov.*.stripe{size,count}=-1 param")
Reviewed-on: https://review.whamcloud.com/27946
--- fs/lustre/lov/lproc_lov.c | 2 +- fs/lustre/obdclass/obd_config.c | 6 +++--- include/uapi/linux/lustre/lustre_idl.h | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-)
fs/lustre/lov/lproc_lov.c +1 -1
@@ -75,7 +75,7 @@ static ssize_t stripeoffset_show(struct kobject *kobj, struct attribute *attr, obd_kset.kobj);struct lov_desc *desc = &dev->u.lov.desc; -return sprintf (buf,"%llu\n" , desc->ld_default_stripe_offset); +return sprintf (buf,"%lld\n" , desc->ld_default_stripe_offset); } static ssize_t stripeoffset_store(struct kobject *kobj, struct attribute *attr,
fs/lustre/obdclass/obd_config.c +3 -3
@@ -1226,14 +1226,14 @@ int class_process_proc_param(char *prefix, struct lprocfs_vars *lvars,/* rc = -EINVAL; continue parsing other params */ skip++; }else if (rc <0 ) { -CERROR ("%s: error writing proc entry '%s': rc = %d\n" , +CERROR ("%s: error writing parameter '%s': rc = %d\n" , prefix, var->name, rc); rc =0 ; }else { -CDEBUG (D_CONFIG ,"%s.%.*s: Set parameter %.*s=%s\n" , +CDEBUG (D_CONFIG ,"%s.%.*s: set parameter %.*s\n" ,lustre_cfg_string (lcfg,0 ), (int )strlen (prefix) -1 , prefix, - (int )(sval - key -1 ), key, sval); + (int )(sval - key -1 ), key); } }
include/uapi/linux/lustre/lustre_idl.h +3 -3
@@ -2035,10 +2035,10 @@ enum fld_op {struct lov_desc {__u32 ld_tgt_count;/* how many OBD's */ __u32 ld_active_tgt_count;/* how many active */ -__u32 ld_default_stripe_count;/* how many objects are used */ +__s32 ld_default_stripe_count;/* how many objects are used */ __u32 ld_pattern;/* default PATTERN_RAID0 */ -__u64 ld_default_stripe_size;/* in bytes */ -__u64 ld_default_stripe_offset;/* in bytes */ +__u64 ld_default_stripe_size;/* in bytes */ +__s64 ld_default_stripe_offset;/* starting OST index */ __u32 ld_padding_0;/* unused */ __u32 ld_qos_maxage;/* in second */ __u32 ld_padding_1;/* also fix lustre_swab_lov_desc */
Signature · 1 line
1.8.3.1
PATCH 61/151 lustre: brw: add short io osc/ost transfer.
In reply to James SimmonsFrom: Patrick Farrell <[email protected]>
There's no need to do target bulk io for small amount of data, and it requires extra network operations.
For this case we add short i/o. When the i/o size is less than or equal to some number of pages (default 3), we encapsulate the data in the ptlrpc request.
With this patch, 4k direct i/o read latency on a Cray Aries network (data is on flash on another node on the Aries) drops from ~280 microseconds to ~200 microseconds. Write latency drops from ~370 microseconds to ~350 microseconds (much more of write latency is waiting for write commit).
This translates to about a 25-30% performance improvement on 4k direct i/o reads and 4k random reads. (Write performance improvement was small to non-existent.)
Improvement was similar with 8k i/o.
Buffered sequential i/o sees no improvement, because it does not perform small i/os.
Performance data:
access = file-per-process
pattern = segmented (1 segment)
ordering in a file = random offsets
ordering inter file= no tasks offsets
xfersize = 4096 bytes
blocksize = 100 MiB
nprocs xfsize shortio dio random Read (MB/s) 1 4k no yes no 15.0 8 4k no yes no 73.4 16 4k no yes no 81.1 1 4k yes yes no 16.5 8 4k yes yes no 95.2 16 4k yes yes no 107.3 1 4k no no yes 15.5 8 4k no no yes 73.4 16 4k no no yes 81.2 1 4k yes no yes 16.8 8 4k yes no yes 95.0 16 4k yes no yes 106.5
Note even when individual i/o performance is not improved, this change reduces the # of network operations required for small i/o, which can help on large systems.
WC-bug-id: https://jira.whamcloud.com/browse/LU-1757
Cray-bug-id: LUS-187
Lustre-commit: 70f092a05878 ("LU-1757 LU-1757 brw: add short io osc/ost transfer")
Reviewed-on: https://review.whamcloud.com/27767
Lustre-commit: 3483e195314b ("LU-1757 brw: Fix short i/o and enable for mdc")
Reviewed-on: https://review.whamcloud.com/30435
WC-bug-id: https://jira.whamcloud.com/browse/LU-8066
Lustre-commit: 32fb31f3bf3d ("LU-8066 osc: move suitable values from procfs to sysfs")
Reviewed-on: https://review.whamcloud.com/30962
--- fs/lustre/include/lprocfs_status.h | 4 ++ fs/lustre/include/lustre_export.h | 7 ++ fs/lustre/include/lustre_net.h | 37 +++++++++- fs/lustre/include/lustre_osc.h | 12 ++++ fs/lustre/include/lustre_req_layout.h | 1 + fs/lustre/include/obd.h | 1 + fs/lustre/llite/llite_lib.c | 6 +- fs/lustre/obdclass/lprocfs_status.c | 55 +++++++++++++++ fs/lustre/osc/lproc_osc.c | 3 + fs/lustre/osc/osc_page.c | 48 ++++++++++--- fs/lustre/osc/osc_request.c | 129 ++++++++++++++++++++++++++++------ fs/lustre/ptlrpc/layout.c | 10 ++- 12 files changed, 272 insertions(+), 41 deletions(-)
fs/lustre/include/lprocfs_status.h +4 -0
@@ -590,6 +590,10 @@ ssize_t max_pages_per_rpc_show(struct kobject *kobj, struct attribute *attr, char *buf);ssize_t max_pages_per_rpc_store (struct kobject *kobj,struct attribute *attr,const char *buffer,size_t count); +ssize_t short_io_bytes_show (struct kobject *kobj,struct attribute *attr, +char *buf); +ssize_t short_io_bytes_store (struct kobject *kobj,struct attribute *attr, +const char *buffer,size_t count);struct root_squash_info ; int lprocfs_wr_root_squash(const char __user *buffer, unsigned long count,
fs/lustre/include/lustre_export.h +7 -0
@@ -245,6 +245,13 @@ static inline bool imp_connect_disp_stripe(struct obd_import *imp)return ocd->ocd_connect_flags &OBD_CONNECT_DISP_STRIPE ; } +static inline bool imp_connect_shortio (struct obd_import *imp) +{ +struct obd_connect_data *ocd = &imp->imp_connect_data; + +return ocd->ocd_connect_flags &OBD_CONNECT_SHORTIO ; +} +static inline int exp_connect_lockahead_old (struct obd_export *exp) {return !!(exp_connect_flags (exp) &OBD_CONNECT_LOCKAHEAD_OLD );
fs/lustre/include/lustre_net.h +34 -3
@@ -273,9 +273,41 @@#define MDS_MAXREQSIZE (5 * 1024)/* >= 4736 */ /** +* OST_IO_MAXREQSIZE ~= +* lustre_msg + ptlrpc_body + obdo + obd_ioobj + +* DT_MAX_BRW_PAGES * niobuf_remote +* +* - single object with 16 pages is 512 bytes +* - OST_IO_MAXREQSIZE must be at least 1 page of cookies plus some spillover +* - Must be a multiple of 1024 +*/ +#define _OST_MAXREQSIZE_BASE (sizeof(struct lustre_msg) + \ + sizeof(struct ptlrpc_body) + \ + sizeof(struct obdo) + \ + sizeof(struct obd_ioobj) + \ + sizeof(struct niobuf_remote)) +#define _OST_MAXREQSIZE_SUM (_OST_MAXREQSIZE_BASE + \ + sizeof(struct niobuf_remote) * \ + (DT_MAX_BRW_PAGES - 1)) + +/** * FIEMAP request can be 4K+ for now */ #define OST_MAXREQSIZE (16 * 1024) +#define OST_IO_MAXREQSIZE max_t(int, OST_MAXREQSIZE, \ + (((_OST_MAXREQSIZE_SUM - 1) | (1024 - 1)) + 1)) + +/* Safe estimate of free space in standard RPC, provides upper limit for # of +* bytes of i/o to pack in RPC (skipping bulk transfer). +*/ +#define OST_SHORT_IO_SPACE (OST_IO_MAXREQSIZE - _OST_MAXREQSIZE_BASE) + +/* Actual size used for short i/o buffer. Calculation means this: +* At least one page (for large PAGE_SIZE), or 16 KiB, but not more +* than the available space aligned to a page boundary. +*/ +#define OBD_MAX_SHORT_IO_BYTES (min(max(PAGE_SIZE, 16UL * 1024UL), \ + OST_SHORT_IO_SPACE & PAGE_MASK))/* Macro to hide a typecast. */ #define ptlrpc_req_async_args (req) ((void *)&req->rq_async_args)
@@ -1758,13 +1790,12 @@ static inline int ptlrpc_client_bulk_active(struct ptlrpc_request *req)int rc; desc = req->rq_bulk; +if (!desc) +return 0 ;if (req->rq_bulk_deadline >ktime_get_real_seconds ())return 1 ; -if (!desc) -return 0 ; -spin_lock (&desc->bd_lock); rc = desc->bd_md_count;spin_unlock (&desc->bd_lock);
fs/lustre/include/lustre_osc.h +12 -0
@@ -535,6 +535,18 @@ struct osc_page {unsigned long ops_submit_time; }; +struct osc_brw_async_args { +struct obdo *aa_oa; +int aa_requested_nob; +int aa_nio_count; +u32 aa_page_count; +int aa_resends; +struct brw_page **aa_ppga; +struct client_obd *aa_cli; +struct list_head aa_oaps; +struct list_head aa_exts; +}; +extern struct kmem_cache *osc_lock_kmem;extern struct kmem_cache *osc_object_kmem;extern struct kmem_cache *osc_thread_kmem;
fs/lustre/include/lustre_req_layout.h +1 -0
@@ -297,6 +297,7 @@ void req_capsule_shrink(struct req_capsule *pill,extern struct req_msg_field RMF_FIEMAP_KEY ;extern struct req_msg_field RMF_FIEMAP_VAL ;extern struct req_msg_field RMF_OST_ID ; +extern struct req_msg_field RMF_SHORT_IO ;/* MGS config read message format */ extern struct req_msg_field RMF_MGS_CONFIG_BODY ;
fs/lustre/include/obd.h +1 -0
@@ -248,6 +248,7 @@ struct client_obd {atomic_t cl_pending_r_pages;u32 cl_max_pages_per_rpc;u32 cl_max_rpcs_in_flight; +u32 cl_short_io_bytes;struct obd_histogram cl_read_rpc_hist;struct obd_histogram cl_write_rpc_hist;struct obd_histogram cl_read_page_hist;
fs/lustre/llite/llite_lib.c +4 -2
@@ -205,7 +205,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)OBD_CONNECT_BULK_MBITS |OBD_CONNECT_CKSUM |OBD_CONNECT_SUBTREE |OBD_CONNECT_MULTIMODRPCS | -OBD_CONNECT_GRANT_PARAM |OBD_CONNECT_FLAGS2 ; +OBD_CONNECT_GRANT_PARAM | +OBD_CONNECT_SHORTIO |OBD_CONNECT_FLAGS2 ; data->ocd_connect_flags2 =0 ;
@@ -396,7 +397,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt)OBD_CONNECT_JOBSTATS |OBD_CONNECT_LVB_TYPE |OBD_CONNECT_LAYOUTLOCK |OBD_CONNECT_PINGLESS |OBD_CONNECT_LFSCK | -OBD_CONNECT_BULK_MBITS |OBD_CONNECT_FLAGS2 ; +OBD_CONNECT_BULK_MBITS |OBD_CONNECT_SHORTIO | +OBD_CONNECT_FLAGS2 ;/* The client currently advertises support for OBD_CONNECT_LOCKAHEAD_OLD * so it can interoperate with an older version of lockahead which was
fs/lustre/obdclass/lprocfs_status.c +55 -0
58 lines not on this page · raw message
fs/lustre/osc/lproc_osc.c +3 -0
@@ -573,7 +573,9 @@ static ssize_t destroys_in_flight_show(struct kobject *kobj, atomic_read(&obd->u.cli.cl_destroy_in_flight)); }LUSTRE_RO_ATTR (destroys_in_flight); +LUSTRE_RW_ATTR (max_pages_per_rpc); +LUSTRE_RW_ATTR (short_io_bytes);static int osc_unstable_stats_seq_show (struct seq_file *m,void *v) {
@@ -807,6 +809,7 @@ void lproc_osc_attach_seqstat(struct obd_device *dev) &lustre_attr_max_dirty_mb.attr, &lustre_attr_max_pages_per_rpc.attr, &lustre_attr_max_rpcs_in_flight.attr, + &lustre_attr_short_io_bytes.attr, &lustre_attr_resend_count.attr, &lustre_attr_ost_conn_uuid.attr, NULL,
fs/lustre/osc/osc_page.c +37 -11
96 lines not on this page · raw message
fs/lustre/osc/osc_request.c +106 -23
227 lines not on this page · raw message
fs/lustre/ptlrpc/layout.c +8 -2
@@ -574,12 +574,14 @@ &RMF_OST_BODY , &RMF_OBD_IOOBJ , &RMF_NIOBUF_REMOTE , - &RMF_CAPA1 + &RMF_CAPA1 , + &RMF_SHORT_IO };static const struct req_msg_field *ost_brw_read_server[] = { &RMF_PTLRPC_BODY , - &RMF_OST_BODY + &RMF_OST_BODY , + &RMF_SHORT_IO }; static const struct req_msg_field *ost_brw_write_server[] = {
@@ -1102,6 +1104,10 @@ struct req_msg_field RMF_FIEMAP_VAL =DEFINE_MSGF ("fiemap" ,0 ,-1 , lustre_swab_fiemap,NULL );EXPORT_SYMBOL (RMF_FIEMAP_VAL ); +struct req_msg_field RMF_SHORT_IO = +DEFINE_MSGF ("short_io" ,0 ,-1 ,NULL ,NULL ); +EXPORT_SYMBOL (RMF_SHORT_IO ); +struct req_msg_field RMF_HSM_USER_STATE =DEFINE_MSGF ("hsm_user_state" ,0 ,sizeof (struct hsm_user_state ), lustre_swab_hsm_user_state,NULL );
Signature · 1 line
1.8.3.1
PATCH 62/151 lustre: lov: take lov layout lock for I/O with ignore_layout
In reply to James SimmonsFrom: Jinshan Xiong <[email protected]>
A rule of thumb for taking lov layout configuration lock is if I/O is initiated from LLITE layer, it should grab the lock. If an I/O starts from the OSC layer, it won't be necessary because if the OSC object exists, layout reconfiguration will move forward.
Right now CIT_MISC + ci_ignore_layout can identify the I/O from the OSC layer, I just use this in lov_io_init() for this purpose. In the future, an explicit bit may be defined for this.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9785
Lustre-commit: e43b0e5c0ccb ("LU-9785 lov: take lov layout lock for I/O with ignore_layout")
Reviewed-on: https://review.whamcloud.com/29638
--- fs/lustre/lov/lov_object.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
fs/lustre/lov/lov_object.c +7 -1
@@ -1224,8 +1224,14 @@ int lov_io_init(const struct lu_env *env, struct cl_object *obj,PFID (lu_object_fid (&obj->co_lu)), io, io->ci_type, io->ci_ignore_layout, io->ci_verify_layout); +/* IO type CIT_MISC with ci_ignore_layout set are usually invoked from +* the OSC layer. It shouldn't take lov layout conf lock in that case, +* because as long as the OSC object exists, the layout can't be +* reconfigured. +*/ return LOV_2DISPATCH_MAYLOCK (cl2lov (obj), llo_io_init, - !io->ci_ignore_layout, env, obj, io); + !(io->ci_ignore_layout && io->ci_type ==CIT_MISC ), + env, obj, io); }/**
Signature · 1 line
1.8.3.1
PATCH 63/151 lustre: lov: pack lsm_flags from layout
In reply to James SimmonsFrom: Jinshan Xiong <[email protected]>
Pack lsm_flags from compount layout.
WC-bug-id: https://jira.whamcloud.com/browse/LU-9771
Lustre-commit: 9b74a75ecbdc ("LU-9771 lov: pack lsm_flags from layout")
Reviewed-on: https://review.whamcloud.com/29081
--- fs/lustre/lov/lov_ea.c | 1 + fs/lustre/lov/lov_internal.h | 3 ++- fs/lustre/lov/lov_pack.c | 1 + 3 files changed, 4 insertions(+), 1 deletion(-)
fs/lustre/lov/lov_ea.c +1 -0
@@ -446,6 +446,7 @@ static int lsm_verify_comp_md_v1(struct lov_comp_md_v1 *lcm, lsm->lsm_magic =le32_to_cpu (lcm->lcm_magic); lsm->lsm_layout_gen =le32_to_cpu (lcm->lcm_layout_gen); lsm->lsm_entry_count = entry_count; + lsm->lsm_flags =le16_to_cpu (lcm->lcm_flags); lsm->lsm_is_released = true; lsm->lsm_maxbytes =LLONG_MIN ;
fs/lustre/lov/lov_internal.h +2 -1
@@ -86,8 +86,9 @@ struct lov_stripe_md {struct ost_id lsm_oi;u32 lsm_magic;u32 lsm_layout_gen; -u32 lsm_entry_count; +u16 lsm_flags;bool lsm_is_released; +u32 lsm_entry_count;struct lov_stripe_md_entry *lsm_entries[]; };
fs/lustre/lov/lov_pack.c +1 -0
@@ -185,6 +185,7 @@ ssize_t lov_lsm_pack(const struct lov_stripe_md *lsm, void *buf, lcmv1->lcm_magic =cpu_to_le32 (lsm->lsm_magic); lcmv1->lcm_size =cpu_to_le32 (lmm_size); lcmv1->lcm_layout_gen =cpu_to_le32 (lsm->lsm_layout_gen); + lcmv1->lcm_flags =cpu_to_le16 (lsm->lsm_flags); lcmv1->lcm_entry_count =cpu_to_le16 (lsm->lsm_entry_count); offset =sizeof (*lcmv1) +sizeof (*lcme) * lsm->lsm_entry_count;
Signature · 1 line
1.8.3.1