staging/lustre: patch bomb 1
cover staging/lustre: patch bomb 1
[sadly git-send-email died sending the patchset... so resending]
Hi Greg,
Here are 40 patches. Three cleanup patches and 37 patches ported from Lustre master to commit
LU-2940 llite: Fix for oops in vvp_pgcache_show()
There are two more patchsets that I will send out following up.
Thanks,
Tao
Amir Shehata (2):
staging/lustre/lnet: Fix assert on empty group in selftest module staging/lustre/ptlrpc: Fix a crash when dereferencing NULL pointer
Andreas Dilger (2):
staging/lustre/ldlm: fix resource/fid check, use DLDLMRES staging/lustre/idl: remove LASSERT/CLASSERT from lustre_idl.h
Andrew Perepechko (1):
staging/lustre/llite: extended attribute cache
Andriy Skulysh (2):
staging/lustre/ptlrpc: Fix race during exp_flock_hash creation staging/lustre/ptlrpc: flock deadlock detection does not work
Artem Blagodarenko (1):
staging/lustre/mgs: set_param -P option that sets value permanently
Dmitry Eremin (2):
staging/lustre/build: clean up unused variables and dead code staging/lustre/build: fix compilation issue with is_compat_task
Doug Oucharek (1):
staging/lustre/lnet: Add LNet Router Priority parameter
Fan Yong (2):
staging/lustre/scrub: OI scrub on OST staging/lustre/scrub: control OI scrub on OST from user space
JC Lafoucriere (5):
staging/lustre/llite: Access to released file trigs a restore staging/lustre/mdt: HSM coordinator client interface staging/lustre/mdt: HSM coordinator agent interface staging/lustre/api: HSM import uses new released pattern staging/lustre/utils: HSM Posix CopyTool
James Simmons (2):
staging/lustre/autoconf: remove vectored fops tests staging/lustre/autoconf: remove LIBCFS_HAVE_IS_COMPAT_TASK test
Jinshan Xiong (2):
staging/lustre/hsm: Implementation of exclusive open staging/lustre/hsm: Add hsm_release feature.
John L. Hammond (7):
staging/lustre: validate open handle cookies staging/lustre/llite: use correct FID in ll_och_fill() staging/lustre/lov: convert magic to host-endian in lov_dump_lmm() staging/lustre/mdc: prevent fall through in mdc_iocontrol() staging/lustre/lu: shrink lu_object by 8 bytes on x86_64 staging/lustre/llite: don't check for O_CREAT in it_create_mode staging/lustre/llite: pass correct pointer to obd_iocontrol()
Lai Siyao (1):
staging/lustre/llite: remove ll_d_root_ops
Li Xi (1):
staging/lustre/llog: fix return value of llog_alloc_handle
Mikhail Pershin (4):
staging/lustre/server: use unified request handler for MGS staging/lustre/llog: MGC to use OSD API for backup logs staging/lustre/target: move OUT to the unified target code staging/lustre/seq: unified SEQ handler
Nathaniel Clark (1):
staging/lustre/xattr: separate ACL and XATTR caches
Patrick Farrell (1):
staging/lustre/nfs: writing to new files will return ENOENT
Peng Tao (3):
staging/lustre/llite: restore ll_fiemap
staging/lustre: remove lu_target.h
staging/lustre: remove llog_server.c
.../staging/lustre/include/linux/libcfs/curproc.h | 1 - .../staging/lustre/include/linux/libcfs/libcfs.h | 2 - .../lustre/include/linux/libcfs/libcfs_ioctl.h | 1 + .../staging/lustre/include/linux/lnet/lib-lnet.h | 5 +- .../staging/lustre/include/linux/lnet/lib-types.h | 2 +- drivers/staging/lustre/lnet/lnet/api-ni.c | 5 +- drivers/staging/lustre/lnet/lnet/config.c | 39 +- drivers/staging/lustre/lnet/lnet/lib-move.c | 6 + drivers/staging/lustre/lnet/lnet/router.c | 19 +- drivers/staging/lustre/lnet/lnet/router_proc.c | 16 +- drivers/staging/lustre/lnet/selftest/conctl.c | 56 +- drivers/staging/lustre/lnet/selftest/conrpc.c | 2 +- drivers/staging/lustre/lnet/selftest/console.c | 105 ++-- drivers/staging/lustre/lnet/selftest/console.h | 6 +- drivers/staging/lustre/lnet/selftest/rpc.c | 2 - drivers/staging/lustre/lnet/selftest/selftest.h | 3 - drivers/staging/lustre/lnet/selftest/timer.c | 6 +- drivers/staging/lustre/lustre/include/cl_object.h | 6 +- drivers/staging/lustre/lustre/include/dt_object.h | 2 +- .../lustre/lustre/include/linux/lustre_compat25.h | 4 +- .../lustre/lustre/include/linux/lustre_intent.h | 2 +- .../lustre/lustre/include/linux/lustre_lite.h | 1 + drivers/staging/lustre/lustre/include/lu_object.h | 19 - drivers/staging/lustre/lustre/include/lu_target.h | 91 --- .../lustre/lustre/include/lustre/lustre_idl.h | 111 ++-- .../lustre/lustre/include/lustre/lustre_user.h | 63 +- .../lustre/lustre/include/lustre/lustreapi.h | 208 ++++--- drivers/staging/lustre/lustre/include/lustre_cfg.h | 2 + .../staging/lustre/lustre/include/lustre_disk.h | 2 + .../lustre/lustre/include/lustre_dlm_flags.h | 24 +- drivers/staging/lustre/lustre/include/lustre_fid.h | 6 - drivers/staging/lustre/lustre/include/lustre_ha.h | 3 - .../staging/lustre/lustre/include/lustre_handles.h | 5 +- drivers/staging/lustre/lustre/include/lustre_lib.h | 11 +- drivers/staging/lustre/lustre/include/lustre_log.h | 13 +- drivers/staging/lustre/lustre/include/lustre_net.h | 19 +- .../lustre/lustre/include/lustre_req_layout.h | 7 + drivers/staging/lustre/lustre/include/md_object.h | 4 +- drivers/staging/lustre/lustre/include/obd.h | 15 +- drivers/staging/lustre/lustre/include/obd_class.h | 9 +- .../staging/lustre/lustre/include/obd_support.h | 10 + drivers/staging/lustre/lustre/lclient/lcommon_cl.c | 6 + .../staging/lustre/lustre/lclient/lcommon_misc.c | 4 +- drivers/staging/lustre/lustre/ldlm/ldlm_flock.c | 69 ++- drivers/staging/lustre/lustre/ldlm/ldlm_lock.c | 7 +- drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 47 +- drivers/staging/lustre/lustre/ldlm/ldlm_request.c | 39 +- drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 19 +- .../lustre/lustre/libcfs/linux/linux-curproc.c | 13 - drivers/staging/lustre/lustre/libcfs/workitem.c | 55 +- drivers/staging/lustre/lustre/llite/Makefile | 2 +- drivers/staging/lustre/lustre/llite/dcache.c | 77 +-- drivers/staging/lustre/lustre/llite/dir.c | 47 +- drivers/staging/lustre/lustre/llite/file.c | 622 ++++++++++++++++++-- .../staging/lustre/lustre/llite/llite_internal.h | 59 +- drivers/staging/lustre/lustre/llite/llite_lib.c | 91 ++- drivers/staging/lustre/lustre/llite/llite_nfs.c | 6 +- drivers/staging/lustre/lustre/llite/lproc_llite.c | 36 ++ drivers/staging/lustre/lustre/llite/namei.c | 53 +- drivers/staging/lustre/lustre/llite/statahead.c | 7 +- drivers/staging/lustre/lustre/llite/super25.c | 4 + drivers/staging/lustre/lustre/llite/vvp_io.c | 54 +- drivers/staging/lustre/lustre/llite/vvp_object.c | 2 +- drivers/staging/lustre/lustre/llite/xattr.c | 105 ++-- drivers/staging/lustre/lustre/llite/xattr_cache.c | 559 ++++++++++++++++++ .../staging/lustre/lustre/lov/lov_cl_internal.h | 16 + drivers/staging/lustre/lustre/lov/lov_internal.h | 4 +- drivers/staging/lustre/lustre/lov/lov_io.c | 15 +- drivers/staging/lustre/lustre/lov/lov_object.c | 35 +- drivers/staging/lustre/lustre/lov/lov_pack.c | 20 +- drivers/staging/lustre/lustre/mdc/mdc_internal.h | 5 +- drivers/staging/lustre/lustre/mdc/mdc_lib.c | 31 +- drivers/staging/lustre/lustre/mdc/mdc_locks.c | 104 +++- drivers/staging/lustre/lustre/mdc/mdc_reint.c | 2 +- drivers/staging/lustre/lustre/mdc/mdc_request.c | 116 ++-- drivers/staging/lustre/lustre/mgc/libmgc.c | 3 - drivers/staging/lustre/lustre/mgc/mgc_request.c | 492 ++++++++++------ drivers/staging/lustre/lustre/obdclass/genops.c | 2 +- drivers/staging/lustre/lustre/obdclass/llog.c | 214 ++++--- .../staging/lustre/lustre/obdclass/local_storage.c | 9 +- .../staging/lustre/lustre/obdclass/local_storage.h | 3 + .../lustre/lustre/obdclass/lprocfs_status.c | 5 +- drivers/staging/lustre/lustre/obdclass/lu_object.c | 26 +- .../lustre/lustre/obdclass/lustre_handles.c | 4 +- drivers/staging/lustre/lustre/obdclass/md_attrs.c | 4 +- .../staging/lustre/lustre/obdclass/obd_config.c | 50 +- drivers/staging/lustre/lustre/obdclass/obd_mount.c | 3 + .../staging/lustre/lustre/obdecho/echo_client.c | 2 +- drivers/staging/lustre/lustre/ptlrpc/Makefile | 2 +- drivers/staging/lustre/lustre/ptlrpc/client.c | 10 +- drivers/staging/lustre/lustre/ptlrpc/import.c | 2 - drivers/staging/lustre/lustre/ptlrpc/layout.c | 63 +- drivers/staging/lustre/lustre/ptlrpc/llog_server.c | 450 -------------- .../staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c | 4 +- drivers/staging/lustre/lustre/ptlrpc/niobuf.c | 16 +- .../staging/lustre/lustre/ptlrpc/pack_generic.c | 59 +- drivers/staging/lustre/lustre/ptlrpc/pinger.c | 65 -- drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c | 44 +- drivers/staging/lustre/lustre/ptlrpc/wiretest.c | 60 +- 99 files changed, 3043 insertions(+), 1793 deletions(-) delete mode 100644 drivers/staging/lustre/lustre/include/lu_target.h create mode 100644 drivers/staging/lustre/lustre/llite/xattr_cache.c delete mode 100644 drivers/staging/lustre/lustre/ptlrpc/llog_server.c
Signature · 1 line
1.7.9.5
PATCH 1/40 staging/lustre/llite: restore ll_fiemap
From: Peng Tao <[email protected]>
It was removed by coan by mistake when first porting the code.
--- drivers/staging/lustre/lustre/llite/file.c | 33 ++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
drivers/staging/lustre/lustre/llite/file.c +33 -0
@@ -2628,6 +2628,38 @@ int ll_getattr(struct vfsmount *mnt, struct dentry *de, struct kstat *stat)return ll_getattr_it (mnt, de, &it, stat); } +int ll_fiemap (struct inode *inode,struct fiemap_extent_info *fieinfo, +__u64 start,__u64 len) +{ +int rc; +size_t num_bytes; +struct ll_user_fiemap *fiemap; +unsigned int extent_count = fieinfo->fi_extents_max; + + num_bytes =sizeof (*fiemap) + (extent_count * +sizeof (struct ll_fiemap_extent )); +OBD_ALLOC_LARGE (fiemap, num_bytes); + +if (fiemap ==NULL ) +return -ENOMEM ; + + fiemap->fm_flags = fieinfo->fi_flags; + fiemap->fm_extent_count = fieinfo->fi_extents_max; + fiemap->fm_start = start; + fiemap->fm_length = len; +memcpy (&fiemap->fm_extents[0 ], fieinfo->fi_extents_start, +sizeof (struct ll_fiemap_extent )); + + rc =ll_do_fiemap (inode, fiemap, num_bytes); + + fieinfo->fi_flags = fiemap->fm_flags; + fieinfo->fi_extents_mapped = fiemap->fm_mapped_extents; +memcpy (fieinfo->fi_extents_start, &fiemap->fm_extents[0 ], + fiemap->fm_mapped_extents *sizeof (struct ll_fiemap_extent )); + +OBD_FREE_LARGE (fiemap, num_bytes); +return rc; +}struct posix_acl *ll_get_acl (struct inode *inode,int type) {
@@ -2740,6 +2772,7 @@ struct inode_operations ll_file_inode_operations = { .getxattr = ll_getxattr, .listxattr = ll_listxattr, .removexattr = ll_removexattr, + .fiemap = ll_fiemap, .get_acl = ll_get_acl, };
Signature · 1 line
1.7.9.5
PATCH 3/40 staging/lustre: remove llog_server.c
In reply to Peng TaoFrom: Peng Tao <[email protected]>
It is only used by server.
--- drivers/staging/lustre/lustre/include/lustre_net.h | 9 - drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 39 -- drivers/staging/lustre/lustre/ptlrpc/Makefile | 2 +- drivers/staging/lustre/lustre/ptlrpc/llog_server.c | 450 -------------------- 4 files changed, 1 insertion(+), 499 deletions(-) delete mode 100644 drivers/staging/lustre/lustre/ptlrpc/llog_server.c
drivers/staging/lustre/lustre/include/lustre_net.h +0 -9
@@ -3470,15 +3470,6 @@ static inline void ptlrpc_lprocfs_brw(struct ptlrpc_request *req, int bytes) {}#endif /** @} */ -/* ptlrpc/llog_server.c */ -int llog_origin_handle_open (struct ptlrpc_request *req); -int llog_origin_handle_destroy (struct ptlrpc_request *req); -int llog_origin_handle_prev_block (struct ptlrpc_request *req); -int llog_origin_handle_next_block (struct ptlrpc_request *req); -int llog_origin_handle_read_header (struct ptlrpc_request *req); -int llog_origin_handle_close (struct ptlrpc_request *req); -int llog_origin_handle_cancel (struct ptlrpc_request *req); -/* ptlrpc/llog_client.c */ extern struct llog_operations llog_client_ops;
drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c +0 -39
@@ -597,45 +597,6 @@ static int ldlm_callback_handler(struct ptlrpc_request *req) rc =ldlm_handle_setinfo (req);ldlm_callback_reply (req, rc);return 0 ; -case OBD_LOG_CANCEL :/* remove this eventually - for 1.4.0 compat */ -CERROR ("shouldn't be handling OBD_LOG_CANCEL on DLM thread\n" ); -req_capsule_set (&req->rq_pill, &RQF_LOG_CANCEL ); -if (OBD_FAIL_CHECK (OBD_FAIL_OBD_LOG_CANCEL_NET )) -return 0 ; - rc =llog_origin_handle_cancel (req); -if (OBD_FAIL_CHECK (OBD_FAIL_OBD_LOG_CANCEL_REP )) -return 0 ; -ldlm_callback_reply (req, rc); -return 0 ; -case LLOG_ORIGIN_HANDLE_CREATE : -req_capsule_set (&req->rq_pill, &RQF_LLOG_ORIGIN_HANDLE_CREATE ); -if (OBD_FAIL_CHECK (OBD_FAIL_OBD_LOGD_NET )) -return 0 ; - rc =llog_origin_handle_open (req); -ldlm_callback_reply (req, rc); -return 0 ; -case LLOG_ORIGIN_HANDLE_NEXT_BLOCK : -req_capsule_set (&req->rq_pill, - &RQF_LLOG_ORIGIN_HANDLE_NEXT_BLOCK ); -if (OBD_FAIL_CHECK (OBD_FAIL_OBD_LOGD_NET )) -return 0 ; - rc =llog_origin_handle_next_block (req); -ldlm_callback_reply (req, rc); -return 0 ; -case LLOG_ORIGIN_HANDLE_READ_HEADER : -req_capsule_set (&req->rq_pill, - &RQF_LLOG_ORIGIN_HANDLE_READ_HEADER ); -if (OBD_FAIL_CHECK (OBD_FAIL_OBD_LOGD_NET )) -return 0 ; - rc =llog_origin_handle_read_header (req); -ldlm_callback_reply (req, rc); -return 0 ; -case LLOG_ORIGIN_HANDLE_CLOSE : -if (OBD_FAIL_CHECK (OBD_FAIL_OBD_LOGD_NET )) -return 0 ; - rc =llog_origin_handle_close (req); -ldlm_callback_reply (req, rc); -return 0 ;case OBD_QC_CALLBACK :req_capsule_set (&req->rq_pill, &RQF_QC_CALLBACK );if (OBD_FAIL_CHECK (OBD_FAIL_OBD_QC_CALLBACK_NET ))
drivers/staging/lustre/lustre/ptlrpc/Makefile +1 -1
@@ -10,7 +10,7 @@ ldlm_objs += $(LDLM)ldlm_pool.o ldlm_objs += $(LDLM)interval_tree.o ptlrpc_objs := client.o recover.o connection.o niobuf.o pack_generic.o ptlrpc_objs += events.o ptlrpc_module.o service.o pinger.o -ptlrpc_objs += llog_net.o llog_client.o llog_server.o import.o ptlrpcd.o +ptlrpc_objs += llog_net.o llog_client.o import.o ptlrpcd.o ptlrpc_objs += pers.o lproc_ptlrpc.o wiretest.o layout.o ptlrpc_objs += sec.o sec_bulk.o sec_gc.o sec_config.o sec_lproc.o ptlrpc_objs += sec_null.o sec_plain.o nrs.o nrs_fifo.o
drivers/staging/lustre/lustre/ptlrpc/llog_server.c deleted +0 -450
450 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH 2/40 staging/lustre: remove lu_target.h
In reply to Peng TaoFrom: Peng Tao <[email protected]>
It is only needed by server code.
--- drivers/staging/lustre/lustre/include/lu_target.h | 91 --------------------- 1 file changed, 91 deletions(-) delete mode 100644 drivers/staging/lustre/lustre/include/lu_target.h
drivers/staging/lustre/lustre/include/lu_target.h deleted +0 -91
@@ -1,91 +0,0 @@ -/* -* GPL HEADER START -* -* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 only, -* as published by the Free Software Foundation. -* -* This program is distributed in the hope that it will be useful, but -* WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -* General Public License version 2 for more details (a copy is included -* in the LICENSE file that accompanied this code). -* -* You should have received a copy of the GNU General Public License -* version 2 along with this program; If not, see -* http://www.sun.com/software/products/lustre/docs/GPLv2.pdf -* -* Please contact Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, -* CA 95054 USA or visit www.sun.com if you need additional information or -* have any questions. -* -* GPL HEADER END -*/ -/* -* Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved. -* Use is subject to license terms. -* -* Copyright (c) 2011, 2012, Intel Corporation. -*/ -/* -* This file is part of Lustre, http://www.lustre.org/ -* Lustre is a trademark of Sun Microsystems, Inc. -*/ - -#ifndef _LUSTRE_LU_TARGET_H -#define _LUSTRE_LU_TARGET_H - -#include <dt_object.h> -#include <lustre_disk.h> - -struct lu_target { -struct obd_device *lut_obd; -struct dt_device *lut_bottom; -/** last_rcvd file */ -struct dt_object *lut_last_rcvd; -/* transaction callbacks */ -struct dt_txn_callback lut_txn_cb; -/** server data in last_rcvd file */ -struct lr_server_data lut_lsd; -/** Server last transaction number */ -__u64 lut_last_transno; -/** Lock protecting last transaction number */ -spinlock_t lut_translock; -/** Lock protecting client bitmap */ -spinlock_t lut_client_bitmap_lock; -/** Bitmap of known clients */ -unsigned long *lut_client_bitmap; -}; - -typedef void (*tgt_cb_t )(struct lu_target *lut,__u64 transno, -void *data,int err); -struct tgt_commit_cb { -tgt_cb_t tgt_cb_func; -void *tgt_cb_data; -}; - -void tgt_boot_epoch_update (struct lu_target *lut); -int tgt_last_commit_cb_add (struct thandle *th,struct lu_target *lut, -struct obd_export *exp,__u64 transno); -int tgt_new_client_cb_add (struct thandle *th,struct obd_export *exp); -int tgt_init (const struct lu_env *env,struct lu_target *lut, -struct obd_device *obd,struct dt_device *dt); -void tgt_fini (const struct lu_env *env,struct lu_target *lut); -int tgt_client_alloc (struct obd_export *exp); -void tgt_client_free (struct obd_export *exp); -int tgt_client_del (const struct lu_env *env,struct obd_export *exp); -int tgt_client_add (const struct lu_env *env,struct obd_export *exp,int ); -int tgt_client_new (const struct lu_env *env,struct obd_export *exp); -int tgt_client_data_read (const struct lu_env *env,struct lu_target *tg, -struct lsd_client_data *lcd,loff_t *off,int index); -int tgt_client_data_write (const struct lu_env *env,struct lu_target *tg, -struct lsd_client_data *lcd,loff_t *off,struct thandle *th); -int tgt_server_data_read (const struct lu_env *env,struct lu_target *tg); -int tgt_server_data_write (const struct lu_env *env,struct lu_target *tg, -struct thandle *th); -int tgt_server_data_update (const struct lu_env *env,struct lu_target *tg,int sync); -int tgt_truncate_last_rcvd (const struct lu_env *env,struct lu_target *tg,loff_t off); - -#endif /* __LUSTRE_LU_TARGET_H */
Signature · 1 line
1.7.9.5
PATCH 4/40 staging/lustre/llite: Access to released file trigs a restore
In reply to Peng TaoFrom: JC Lafoucriere <[email protected]>
When a client accesses data in a released file,
or truncate it, client must trig a restore request. During this restore, the client must not glimpse and must use size from MDT. To bring the "restore is running" information on the client we add a new t_state bit field to mdt_info which will be used to carry transient file state. To memorise this information in the inode we add a new flag LLIF_FILE_RESTORING.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3432
Lustre-change: http://review.whamcloud.com/6537
--- drivers/staging/lustre/lustre/include/cl_object.h | 6 ++- .../lustre/lustre/include/lustre/lustre_idl.h | 14 +++-- drivers/staging/lustre/lustre/lclient/lcommon_cl.c | 6 +++ drivers/staging/lustre/lustre/llite/file.c | 40 ++++++++++++++- .../staging/lustre/lustre/llite/llite_internal.h | 3 ++ drivers/staging/lustre/lustre/llite/llite_lib.c | 39 +++++++++++++- drivers/staging/lustre/lustre/llite/vvp_io.c | 54 ++++++++++++++++++-- drivers/staging/lustre/lustre/lov/lov_io.c | 15 ++++-- .../staging/lustre/lustre/ptlrpc/pack_generic.c | 52 +++++++++---------- drivers/staging/lustre/lustre/ptlrpc/wiretest.c | 14 ++--- 10 files changed, 194 insertions(+), 49 deletions(-)
drivers/staging/lustre/lustre/include/cl_object.h +5 -1
@@ -2388,7 +2388,11 @@ struct cl_io {* Right now, only two opertaions need to verify layout: glimpse * and setattr. */ - ci_verify_layout:1 ; + ci_verify_layout:1 , +/** +* file is released, restore has to to be triggered by vvp layer +*/ + ci_restore_needed:1;/** * Number of pages owned by this IO. For invariant checking. */
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +9 -5
@@ -1725,10 +1725,7 @@ static inline __u32 lov_mds_md_size(__u16 stripes, __u32 lmm_magic)#define OBD_MD_MDS (0x0000000100000000ULL)/* where an inode lives on */ #define OBD_MD_REINT (0x0000000200000000ULL)/* reintegrate oa */ #define OBD_MD_MEA (0x0000000400000000ULL)/* CMD split EA */ - -/* OBD_MD_MDTIDX is used to get MDT index, but it is never been used overwire, -* and it is already obsolete since 2.3 */ -/* #define OBD_MD_MDTIDX (0x0000000800000000ULL) */ +#define OBD_MD_TSTATE (0x0000000800000000ULL)/* transient state field */ #define OBD_MD_FLXATTR (0x0000001000000000ULL)/* xattr */ #define OBD_MD_FLXATTRLS (0x0000002000000000ULL)/* xattr list */
@@ -2207,6 +2204,11 @@ static inline int ll_inode_to_ext_flags(int iflags) ((iflags & S_IMMUTABLE) ? LUSTRE_IMMUTABLE_FL : 0)); } +/* 64 possible states */ +enum md_transient_state { +MS_RESTORE = (1 <<0 ),/* restore is running */ +}; +struct mdt_body {struct lu_fid fid1;struct lu_fid fid2;
@@ -2218,7 +2220,9 @@ struct mdt_body {obd_time ctime;__u64 blocks;/* XID, in the case of MDS_READPAGE */ __u64 ioepoch; -__u64 unused1;/* was "ino" until 2.4.0 */ +__u64 t_state;/* transient file state defined in +* enum md_transient_state +* was "ino" until 2.4.0 */ __u32 fsuid;__u32 fsgid;__u32 capability;
drivers/staging/lustre/lustre/lclient/lcommon_cl.c +6 -0
@@ -1006,6 +1006,12 @@ again:cl_io_fini (env, io);if (unlikely (io->ci_need_restart)) goto again; +/* HSM import case: file is released, cannot be restored +* no need to fail except if restore registration failed +* with -ENODATA */ +if (result == -ENODATA && io->ci_restore_needed && + io->ci_result != -ENODATA ) + result =0 ;cl_env_put (env, &refcheck);return result; }
drivers/staging/lustre/lustre/llite/file.c +38 -2
@@ -905,7 +905,7 @@ out:cl_io_fini (env, io);/* If any bit been read/written (result != 0), we just return * short read/write instead of restart io. */ -if (result ==0 && io->ci_need_restart) { +if ((result ==0 || result == -ENODATA ) && io->ci_need_restart) {CDEBUG (D_VFSTRACE ,"Restart %s on %s from %lld, count:%zd\n" , iot ==CIT_READ ?"read" :"write" , file->f_dentry->d_name.name, *ppos, count);
@@ -2581,7 +2581,15 @@ int ll_inode_revalidate_it(struct dentry *dentry, struct lookup_intent *it,LTIME_S (inode->i_mtime) =ll_i2info (inode)->lli_lvb.lvb_mtime;LTIME_S (inode->i_ctime) =ll_i2info (inode)->lli_lvb.lvb_ctime; }else { - rc =ll_glimpse_size (inode); +/* In case of restore, the MDT has the right size and has +* already send it back without granting the layout lock, +* inode is up-to-date so glimpse is useless. +* Also to glimpse we need the layout, in case of a running +* restore the MDT holds the layout lock so the glimpse will +* block up to the end of restore (getattr will block) +*/ +if (!(ll_i2info (inode)->lli_flags &LLIF_FILE_RESTORING )) + rc =ll_glimpse_size (inode); }return rc; }
@@ -3006,6 +3014,7 @@ static int ll_layout_lock_set(struct lustre_handle *lockh, ldlm_mode_t mode,unlock_res_and_lock (lock);/* checking lvb_ready is racy but this is okay. The worst case is * that multi processes may configure the file on the same time. */ +if (lvb_ready || !reconf) { rc = -ENODATA ;if (lvb_ready) {
@@ -3183,3 +3192,30 @@ again:return rc; } + +/** +* This function send a restore request to the MDT +*/ +int ll_layout_restore (struct inode *inode) +{ +struct hsm_user_request *hur; +int len, rc; + + len =sizeof (struct hsm_user_request ) + +sizeof (struct hsm_user_item ); +OBD_ALLOC (hur, len); +if (hur ==NULL ) +return -ENOMEM ; + + hur->hur_request.hr_action =HUA_RESTORE ; + hur->hur_request.hr_archive_id =0 ; + hur->hur_request.hr_flags =0 ; +memcpy (&hur->hur_user_item[0 ].hui_fid, &ll_i2info (inode)->lli_fid, +sizeof (hur->hur_user_item[0 ].hui_fid)); + hur->hur_user_item[0 ].hui_extent.length =-1 ; + hur->hur_request.hr_itemcount =1 ; + rc =obd_iocontrol (LL_IOC_HSM_REQUEST ,cl_i2sbi (inode)->ll_md_exp, + len, hur,NULL ); +OBD_FREE (hur, len); +return rc; +}
drivers/staging/lustre/lustre/llite/llite_internal.h +3 -0
@@ -124,6 +124,8 @@ enum lli_flags {LLIF_SRVLOCK = (1 <<5 ),/* File data is modified. */ LLIF_DATA_MODIFIED = (1 <<6 ), +/* File is being restored */ + LLIF_FILE_RESTORING = (1 << 7), }; struct ll_inode_info {
@@ -1578,5 +1580,6 @@ enum {int ll_layout_conf (struct inode *inode,const struct cl_object_conf *conf);int ll_layout_refresh (struct inode *inode,__u32 *gen); +int ll_layout_restore (struct inode *inode);#endif /* LLITE_INTERNAL_H */
drivers/staging/lustre/lustre/llite/llite_lib.c +38 -1
@@ -1353,6 +1353,7 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr)struct ll_inode_info *lli =ll_i2info (inode);struct md_op_data *op_data =NULL ;struct md_open_data *mod =NULL ; +bool file_is_released = false;int rc =0 , rc1 =0 ; CDEBUG(D_VFSTRACE, "%s: setattr inode %p/fid:"DFID" from %llu to %llu, "
@@ -1436,10 +1437,40 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr) (attr->ia_valid & (ATTR_SIZE | ATTR_MTIME | ATTR_MTIME_SET))) op_data->op_flags = MF_EPOCH_OPEN; +/* truncate on a released file must failed with -ENODATA, +* so size must not be set on MDS for released file +* but other attributes must be set +*/ +if (S_ISREG (inode->i_mode)) { +struct lov_stripe_md *lsm; +__u32 gen; + +ll_layout_refresh (inode, &gen); + lsm =ccc_inode_lsm_get (inode); +if (lsm && lsm->lsm_pattern &LOV_PATTERN_F_RELEASED ) + file_is_released = true; +ccc_inode_lsm_put (inode, lsm); + } + +/* clear size attr for released file +* we clear the attribute send to MDT in op_data, not the original +* received from caller in attr which is used later to +* decide return code */ +if (file_is_released && (attr->ia_valid &ATTR_SIZE )) + op_data->op_attr.ia_valid &= ~ATTR_SIZE ; + rc =ll_md_setattr (dentry, op_data, &mod);if (rc)GOTO (out, rc); +/* truncate failed, others succeed */ +if (file_is_released) { +if (attr->ia_valid &ATTR_SIZE ) +GOTO (out, rc = -ENODATA ); +else +GOTO (out, rc =0 ); + } +/* RPC to MDT is sent, cancel data modification flag */ if (rc ==0 && (op_data->op_bias &MDS_DATA_MODIFIED )) {spin_lock (&lli->lli_lock);
@@ -1453,7 +1484,7 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr)if (attr->ia_valid & (ATTR_SIZE |ATTR_ATIME |ATTR_ATIME_SET | -ATTR_MTIME |ATTR_MTIME_SET )) +ATTR_MTIME |ATTR_MTIME_SET )) {/* For truncate and utimes sending attributes to OSTs, setting * mtime/atime to the past will be performed under PW [0:EOF] * extent lock (new_size:EOF for truncate). It may seem
@@ -1461,6 +1492,7 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr)* setting times to past, but it is necessary due to possible * time de-synchronization between MDT inode and OST objects */ rc =ll_setattr_ost (inode, attr); + } out:if (op_data) {if (op_data->op_ioepoch) {
@@ -1761,6 +1793,11 @@ void ll_update_inode(struct inode *inode, struct lustre_md *md)LASSERT (md->oss_capa);ll_add_capa (inode, md->oss_capa); } + +if (body->valid &OBD_MD_TSTATE ) { +if (body->t_state &MS_RESTORE ) + lli->lli_flags |=LLIF_FILE_RESTORING ; + } }void ll_read_inode2 (struct inode *inode,void *opaque)
drivers/staging/lustre/lustre/llite/vvp_io.c +49 -5
72 lines not on this page · raw message
drivers/staging/lustre/lustre/lov/lov_io.c +12 -3
@@ -947,14 +947,23 @@ int lov_io_init_released(const struct lu_env *env, struct cl_object *obj,LASSERTF (0 ,"invalid type %d\n" , io->ci_type);case CIT_MISC :case CIT_FSYNC : - result =+1 ; + result =1 ;break ;case CIT_SETATTR : +/* the truncate to 0 is managed by MDT: +* - in open, for open O_TRUNC +* - in setattr, for truncate +*/ +/* the truncate is for size > 0 so triggers a restore */ +if (cl_io_is_trunc (io)) + io->ci_restore_needed =1 ; + result = -ENODATA ; +break ;case CIT_READ :case CIT_WRITE :case CIT_FAULT : -/* TODO: need to restore the file. */ - result = -EBADF ; + io->ci_restore_needed =1 ; + result = -ENODATA ;break ; }if (result ==0 ) {
drivers/staging/lustre/lustre/ptlrpc/pack_generic.c +26 -26
60 lines not on this page · raw message
drivers/staging/lustre/lustre/ptlrpc/wiretest.c +8 -6
@@ -49,8 +49,8 @@ void lustre_assert_wire_constants(void) {/* Wire protocol assertions generated by 'wirecheck' * (make -C lustre/utils newwiretest) -* running on Linux deva 2.6.32.279.lustre #5 SMP Tue Apr 9 22:52:17 CST 2013 x86_64 x86_64 x -* with gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) */ +* running on Linux centos6-bis 2.6.32-358.0.1.el6-head #3 SMP Wed Apr 17 17:37:43 CEST 2013 +* with gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) */ /* Constants... */
@@ -1335,6 +1335,8 @@ void lustre_assert_wire_constants(void) OBD_MD_REINT);LASSERTF (OBD_MD_MEA == (0x0000000400000000ULL ),"found 0x%.16llxULL\n" ,OBD_MD_MEA ); +LASSERTF (OBD_MD_TSTATE == (0x0000000800000000ULL ),"found 0x%.16llxULL\n" , +OBD_MD_TSTATE );LASSERTF (OBD_MD_FLXATTR == (0x0000001000000000ULL ),"found 0x%.16llxULL\n" ,OBD_MD_FLXATTR ); LASSERTF(OBD_MD_FLXATTRLS == (0x0000002000000000ULL), "found 0x%.16llxULL\n",
@@ -1918,10 +1920,10 @@ void lustre_assert_wire_constants(void) (long long)(int)offsetof(struct mdt_body, blocks));LASSERTF ((int )sizeof (((struct mdt_body *)0 )->blocks) ==8 ,"found %lld\n" , (long long )(int )sizeof (((struct mdt_body *)0 )->blocks)); -LASSERTF ((int )offsetof(struct mdt_body , unused1) ==96 ,"found %lld\n" , - (long long )(int )offsetof(struct mdt_body , unused1)); -LASSERTF ((int )sizeof (((struct mdt_body *)0 )->unused1) ==8 ,"found %lld\n" , - (long long )(int )sizeof (((struct mdt_body *)0 )->unused1)); +LASSERTF ((int )offsetof(struct mdt_body , t_state) ==96 ,"found %lld\n" , + (long long )(int )offsetof(struct mdt_body , t_state)); +LASSERTF ((int )sizeof (((struct mdt_body *)0 )->t_state) ==8 ,"found %lld\n" , + (long long )(int )sizeof (((struct mdt_body *)0 )->t_state));LASSERTF ((int )offsetof(struct mdt_body , fsuid) ==104 ,"found %lld\n" , (long long )(int )offsetof(struct mdt_body , fsuid)); LASSERTF((int)sizeof(((struct mdt_body *)0)->fsuid) == 4, "found %lld\n",
Signature · 1 line
1.7.9.5
PATCH 6/40 staging/lustre/llite: use correct FID in ll_och_fill()
In reply to Peng TaoFrom: "John L. Hammond" <[email protected]>
When ll_intent_file_open() is called on a file with a stale dentry, ll_och_fill() may incorrectly use the FID from the struct ll_inode_info rather than the FID from the response body (which is the correct FID for the close). Fix this, remove the ll_inode_info parameter from ll_och_fill(), and move the call to ll_ioepoch_open() from ll_och_fill() to ll_local_open().
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3233
Lustre-change: http://review.whamcloud.com/6695
--- drivers/staging/lustre/lustre/llite/file.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-)
drivers/staging/lustre/lustre/llite/file.c +8 -17
@@ -431,22 +431,17 @@ void ll_ioepoch_open(struct ll_inode_info *lli, __u64 ioepoch) } } -static int ll_och_fill (struct obd_export *md_exp,struct ll_inode_info *lli, -struct lookup_intent *it,struct obd_client_handle *och) +static int ll_och_fill (struct obd_export *md_exp,struct lookup_intent *it, +struct obd_client_handle *och) {struct ptlrpc_request *req = it->d.lustre.it_data;struct mdt_body *body; -LASSERT (och); - body =req_capsule_server_get (&req->rq_pill, &RMF_MDT_BODY ); -LASSERT (body !=NULL );/* reply already checked out */ - -memcpy (&och->och_fh, &body->handle,sizeof (body->handle)); + och->och_fh = body->handle; + och->och_fid = body->fid1; och->och_magic =OBD_CLIENT_HANDLE_MAGIC ; - och->och_fid = lli->lli_fid; och->och_flags = it->it_flags; -ll_ioepoch_open (lli, body->ioepoch);return md_set_open_replay_data (md_exp, och, req); }
@@ -466,15 +461,12 @@ int ll_local_open(struct file *file, struct lookup_intent *it,struct mdt_body *body;int rc; - rc =ll_och_fill (ll_i2sbi (inode)->ll_md_exp, lli, it, och); -if (rc) + rc =ll_och_fill (ll_i2sbi (inode)->ll_md_exp, it, och); +if (rc !=0 )return rc; body =req_capsule_server_get (&req->rq_pill, &RMF_MDT_BODY ); -if ((it->it_flags &FMODE_WRITE ) && - (body->valid &OBD_MD_FLSIZE )) -CDEBUG (D_INODE ,"Epoch " LPU64 " opened on " DFID "\n" , - lli->lli_ioepoch,PFID (&lli->lli_fid)); +ll_ioepoch_open (lli, body->ioepoch); }LUSTRE_FPRIVATE (file) = fd;
@@ -1482,8 +1474,7 @@ int ll_release_openhandle(struct dentry *dentry, struct lookup_intent *it)if (!och)GOTO (out, rc = -ENOMEM ); -ll_och_fill (ll_i2sbi (inode)->ll_md_exp, -ll_i2info (inode), it, och); +ll_och_fill (ll_i2sbi (inode)->ll_md_exp, it, och); rc =ll_close_inode_openhandle (ll_i2sbi (inode)->ll_md_exp, inode, och);
Signature · 1 line
1.7.9.5
PATCH 5/40 staging/lustre: validate open handle cookies
In reply to Peng TaoFrom: "John L. Hammond" <[email protected]>
Add a const void *h_owner member to struct portals_handle. Add a const void *owner parameter to class_handle2object() which must be matched by the h_owner member of the handle in addition to the cookie. Adjust the callers of class_handle2object() accordingly, using NULL as the argument to the owner parameter, except in the case of mdt_handle2mfd() where we add an explicit mdt_export_data parameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to the mdt_export_data into mdt_mfd_new() and store it in h_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3233
Lustre-change: http://review.whamcloud.com/6938
--- .../staging/lustre/lustre/include/lustre_handles.h | 5 +++-- drivers/staging/lustre/lustre/ldlm/ldlm_lock.c | 2 +- drivers/staging/lustre/lustre/lov/lov_internal.h | 4 ++-- drivers/staging/lustre/lustre/obdclass/genops.c | 2 +- .../lustre/lustre/obdclass/lustre_handles.c | 4 ++-- 5 files changed, 9 insertions(+), 8 deletions(-)
drivers/staging/lustre/lustre/include/lustre_handles.h +3 -2
@@ -66,7 +66,8 @@ struct portals_handle_ops {struct portals_handle {struct list_head h_link;__u64 h_cookie; -struct portals_handle_ops *h_ops; +const void *h_owner; +struct portals_handle_ops *h_ops;/* newly added fields to handle the RCU issue. -jxiong */ cfs_rcu_head_t h_rcu;
@@ -83,7 +84,7 @@ void class_handle_hash(struct portals_handle *, struct portals_handle_ops *ops);void class_handle_unhash (struct portals_handle *);void class_handle_hash_back (struct portals_handle *); -void *class_handle2object (__u64 cookie); +void *class_handle2object (__u64 cookie,const void *owner);void class_handle_free_cb (cfs_rcu_head_t *);int class_handle_init (void );void class_handle_cleanup (void );
drivers/staging/lustre/lustre/ldlm/ldlm_lock.c +1 -1
@@ -570,7 +570,7 @@ struct ldlm_lock *__ldlm_handle2lock(const struct lustre_handle *handle,LASSERT (handle); - lock =class_handle2object (handle->cookie); + lock =class_handle2object (handle->cookie,NULL );if (lock ==NULL )return NULL ;
drivers/staging/lustre/lustre/lov/lov_internal.h +2 -2
@@ -107,10 +107,10 @@ static inline void lov_put_reqset(struct lov_request_set *set) }static inline struct lov_lock_handles * -lov_handle2llh (struct lustre_handle *handle) +lov_handle2llh (const struct lustre_handle *handle) {LASSERT (handle !=NULL ); -return (class_handle2object (handle->cookie)); +return class_handle2object (handle->cookie,NULL ); }static inline void lov_llh_put (struct lov_lock_handles *llh)
drivers/staging/lustre/lustre/obdclass/genops.c +1 -1
@@ -701,7 +701,7 @@ struct obd_export *class_conn2export(struct lustre_handle *conn) }CDEBUG (D_INFO ,"looking for export cookie " LPX64 "\n" , conn->cookie); - export =class_handle2object (conn->cookie); + export =class_handle2object (conn->cookie,NULL );return export; }EXPORT_SYMBOL (class_conn2export);
drivers/staging/lustre/lustre/obdclass/lustre_handles.c +2 -2
@@ -147,7 +147,7 @@ void class_handle_hash_back(struct portals_handle *h) }EXPORT_SYMBOL (class_handle_hash_back); -void *class_handle2object (__u64 cookie) +void *class_handle2object (__u64 cookie,const void *owner) {struct handle_bucket *bucket;struct portals_handle *h;
@@ -161,7 +161,7 @@ void *class_handle2object(__u64 cookie)rcu_read_lock ();list_for_each_entry_rcu (h, &bucket->head, h_link) { -if (h->h_cookie != cookie) +if (h->h_cookie != cookie || h->h_owner != owner)continue ;spin_lock (&h->h_lock);
Signature · 1 line
1.7.9.5
PATCH 14/40 staging/lustre/autoconf: remove LIBCFS_HAVE_IS_COMPAT_TASK test
In reply to Peng TaoFrom: James Simmons <[email protected]>
is_compat_task has been defined on all arches since v2.6.29. We can remove the test and dead code.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2800
Lustre-change: http://review.whamcloud.com/5393
--- .../staging/lustre/include/linux/libcfs/curproc.h | 1 - .../lustre/lustre/libcfs/linux/linux-curproc.c | 13 ------------- .../staging/lustre/lustre/llite/llite_internal.h | 3 ++- 3 files changed, 2 insertions(+), 15 deletions(-)
drivers/staging/lustre/include/linux/libcfs/curproc.h +0 -1
@@ -61,7 +61,6 @@ int cfs_curproc_groups_nr(void);*/ /* check if task is running in compat mode.*/ -int current_is_32bit (void );#define current_pid () (current->pid)#define current_comm () (current->comm)int cfs_get_environ (const char *key,char *value,int *val_len);
drivers/staging/lustre/lustre/libcfs/linux/linux-curproc.c +0 -13
@@ -140,18 +140,6 @@ int cfs_capable(cfs_cap_t cap)return capable (cfs_cap_unpack (cap)); } -/* Check if task is running in 32-bit API mode, for the purpose of -* userspace binary interfaces. On 32-bit Linux this is (unfortunately) -* always true, even if the application is using LARGEFILE64 and 64-bit -* APIs, because Linux provides no way for the filesystem to know if it -* is called via 32-bit or 64-bit APIs. Other clients may vary. On -* 64-bit systems, this will only be true if the binary is calling a -* 32-bit system call. */ -int current_is_32bit (void ) -{ -return is_compat_task (); -} -static int cfs_access_process_vm (struct task_struct *tsk,unsigned long addr,void *buf,int len,int write) {
@@ -311,7 +299,6 @@ EXPORT_SYMBOL(cfs_cap_raised);EXPORT_SYMBOL (cfs_curproc_cap_pack);EXPORT_SYMBOL (cfs_curproc_cap_unpack);EXPORT_SYMBOL (cfs_capable); -EXPORT_SYMBOL (current_is_32bit);/* * Local variables:
drivers/staging/lustre/lustre/llite/llite_internal.h +2 -1
@@ -46,6 +46,7 @@#include <lclient.h> #include <lustre_mdc.h> #include <linux/lustre_intent.h> +#include <linux/compat.h> #ifndef FMODE_EXEC #define FMODE_EXEC 0
@@ -649,7 +650,7 @@ static inline int ll_need_32bit_api(struct ll_sb_info *sbi)#if BITS_PER_LONG ==32 return 1 ;#else -return unlikely (current_is_32bit () || (sbi->ll_flags &LL_SBI_32BIT_API )); +return unlikely (is_compat_task () || (sbi->ll_flags &LL_SBI_32BIT_API ));#endif }
Signature · 1 line
1.7.9.5
PATCH 15/40 staging/lustre/llog: fix return value of llog_alloc_handle
In reply to Peng TaoFrom: Li Xi <[email protected]>
llog_open() calls llog_alloc_handle() taking NULL as the error return value. But llog_alloc_handle() returns ERR_PTR(-ENOMEM) instead when error.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3470
Lustre-change: http://review.whamcloud.com/6644
--- drivers/staging/lustre/lustre/obdclass/llog.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/staging/lustre/lustre/obdclass/llog.c +1 -1
@@ -62,7 +62,7 @@ struct llog_handle *llog_alloc_handle(void)OBD_ALLOC_PTR (loghandle);if (loghandle ==NULL ) -return ERR_PTR (-ENOMEM ); +return NULL ;init_rwsem (&loghandle->lgh_lock);spin_lock_init (&loghandle->lgh_hdr_lock);
Signature · 1 line
1.7.9.5
PATCH 16/40 staging/lustre/lov: convert magic to host-endian in lov_dump_lmm()
In reply to Peng TaoFrom: "John L. Hammond" <[email protected]>
In lov_dump_lmm(), convert the lmm_magic from little-endian to host-endian byte order before the switch statement, as the other lov_dump_xxx() and lov_verify_xxx() functions already do. Remove the unused macro LMM_ASSERT().
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3297
Lustre-change: http://review.whamcloud.com/6290
--- drivers/staging/lustre/lustre/lov/lov_pack.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-)
drivers/staging/lustre/lustre/lov/lov_pack.c +9 -11
@@ -105,24 +105,22 @@ void lov_dump_lmm(int level, void *lmm) {int magic; - magic = ((struct lov_mds_md_v1 *)(lmm))->lmm_magic; + magic =le32_to_cpu (((struct lov_mds_md *)lmm)->lmm_magic);switch (magic) {case LOV_MAGIC_V1 : -return lov_dump_lmm_v1 (level, (struct lov_mds_md_v1 *)(lmm)); +lov_dump_lmm_v1 (level, (struct lov_mds_md_v1 *)lmm); +break ;case LOV_MAGIC_V3 : -return lov_dump_lmm_v3 (level, (struct lov_mds_md_v3 *)(lmm)); +lov_dump_lmm_v3 (level, (struct lov_mds_md_v3 *)lmm); +break ;default : -CERROR ("Cannot recognize lmm_magic %x" , magic); +CDEBUG (level,"unrecognized lmm_magic %x, assuming %x\n" , + magic,LOV_MAGIC_V1 ); +lov_dump_lmm_common (level, lmm); +break ; } -return ; } -#define LMM_ASSERT (test) \ -do { \ - if (!(test)) lov_dump_lmm(D_ERROR, lmm); \ - LASSERT(test);/* so we know what assertion failed */ \ -}while (0 ) -/* Pack LOV object metadata for disk storage. It is packed in LE byte * order and is opaque to the networking layer. *
Signature · 1 line
1.7.9.5
PATCH 13/40 staging/lustre/autoconf: remove vectored fops tests
In reply to Peng TaoFrom: James Simmons <[email protected]>
file_operations.readv/writev have been removed since v2.6.19 We can remove the test and the dead code.
Lustre-change: http://review.whamcloud.com/5343
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2800
--- drivers/staging/lustre/lustre/llite/file.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
drivers/staging/lustre/lustre/llite/file.c +6 -11
@@ -3000,17 +3000,12 @@ int ll_inode_permission(struct inode *inode, int mask)return rc; } -#define READ_METHOD aio_read -#define READ_FUNCTION ll_file_aio_read -#define WRITE_METHOD aio_write -#define WRITE_FUNCTION ll_file_aio_write -/* -o localflock - only provides locally consistent flock locks */ struct file_operations ll_file_operations = { .read = ll_file_read, - .READ_METHOD =READ_FUNCTION , + .aio_read = ll_file_aio_read, .write = ll_file_write, - .WRITE_METHOD =WRITE_FUNCTION , + .aio_write = ll_file_aio_write, .unlocked_ioctl = ll_file_ioctl, .open = ll_file_open, .release = ll_file_release,
@@ -3023,9 +3018,9 @@ struct file_operations ll_file_operations = {struct file_operations ll_file_operations_flock = { .read = ll_file_read, - .READ_METHOD =READ_FUNCTION , + .aio_read = ll_file_aio_read, .write = ll_file_write, - .WRITE_METHOD =WRITE_FUNCTION , + .aio_write = ll_file_aio_write, .unlocked_ioctl = ll_file_ioctl, .open = ll_file_open, .release = ll_file_release,
@@ -3041,9 +3036,9 @@ struct file_operations ll_file_operations_flock = {/* These are for -o noflock - to return ENOSYS on flock calls */ struct file_operations ll_file_operations_noflock = { .read = ll_file_read, - .READ_METHOD =READ_FUNCTION , + .aio_read = ll_file_aio_read, .write = ll_file_write, - .WRITE_METHOD =WRITE_FUNCTION , + .aio_write = ll_file_aio_write, .unlocked_ioctl = ll_file_ioctl, .open = ll_file_open, .release = ll_file_release,
Signature · 1 line
1.7.9.5
PATCH 19/40 staging/lustre/lu: shrink lu_object by 8 bytes on x86_64
In reply to Peng TaoFrom: "John L. Hammond" <[email protected]>
Remove the lo_depth member from struct lu_object. This field is never set and only read in lu_object_print(). Remove the lo_flags member. This field was only used in lu_object_alloc() and can be replaced with an on-stack mask to keep trace of which layers have been allocated.
Lustre-change: http://review.whamcloud.com/5890
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3059
--- drivers/staging/lustre/lustre/include/lu_object.h | 19 ---------------- drivers/staging/lustre/lustre/obdclass/lu_object.c | 24 +++++++++++++------- 2 files changed, 16 insertions(+), 27 deletions(-)
drivers/staging/lustre/lustre/include/lu_object.h +0 -19
@@ -398,17 +398,6 @@ static inline int lu_device_is_md(const struct lu_device *d) }/** -* Flags for the object layers. -*/ -enum lu_object_flags { -/** -* this flags is set if lu_object_operations::loo_object_init() has -* been called for this layer. Used by lu_object_alloc(). -*/ -LU_OBJECT_ALLOCATED = (1 <<0 ) -}; - -/** * Common object attributes. */ struct lu_attr {
@@ -486,14 +475,6 @@ struct lu_object {*/ struct list_head lo_linkage;/** -* Depth. Top level layer depth is 0. -*/ -int lo_depth; -/** -* Flags from enum lu_object_flags. -*/ -__u32 lo_flags; -/** * Link to the device, for debugging. */ struct lu_ref_link lo_dev_ref;
drivers/staging/lustre/lustre/obdclass/lu_object.c +16 -8
@@ -200,6 +200,8 @@ static struct lu_object *lu_object_alloc(const struct lu_env *env,struct lu_object *scan;struct lu_object *top;struct list_head *layers; +unsigned int init_mask =0 ; +unsigned int init_flag;int clean;int result;
@@ -218,15 +220,17 @@ static struct lu_object *lu_object_alloc(const struct lu_env *env,*/ top->lo_header->loh_fid = *f; layers = &top->lo_header->loh_layers; + do { /* * Call ->loo_object_init() repeatedly, until no more new * object slices are created. */ clean = 1; + init_flag = 1; list_for_each_entry(scan, layers, lo_linkage) { - if (scan->lo_flags & LU_OBJECT_ALLOCATED) - continue; + if (init_mask & init_flag) + goto next; clean = 0; scan->lo_header = top->lo_header; result = scan->lo_ops->loo_object_init(env, scan, conf);
@@ -234,7 +238,9 @@ static struct lu_object *lu_object_alloc(const struct lu_env *env,lu_object_free (env, top);return ERR_PTR (result); } - scan->lo_flags |=LU_OBJECT_ALLOCATED ; + init_mask |= init_flag; +next: + init_flag <<=1 ; } }while (!clean);
@@ -487,23 +493,25 @@ void lu_object_print(const struct lu_env *env, void *cookie, {static const char ruler[] ="........................................" ;struct lu_object_header *top; -int depth; +int depth =4 ; top = o->lo_header;lu_object_header_print (env, cookie, printer, top); - (*printer)(env, cookie,"{ \n" ); -list_for_each_entry (o, &top->loh_layers, lo_linkage) { - depth = o->lo_depth +4 ; + (*printer)(env, cookie,"{\n" ); +list_for_each_entry (o, &top->loh_layers, lo_linkage) {/* * print `.' \a depth times followed by type name and address */ (*printer)(env, cookie,"%*.*s%s@%p" , depth, depth, ruler, o->lo_dev->ld_type->ldt_name, o); +if (o->lo_ops->loo_object_print !=NULL ) - o->lo_ops->loo_object_print (env, cookie, printer, o); + (*o->lo_ops->loo_object_print)(env, cookie, printer, o); + (*printer)(env, cookie,"\n" ); } + (*printer)(env, cookie,"} header@%p\n" , top); }EXPORT_SYMBOL (lu_object_print);
Signature · 1 line
1.7.9.5
PATCH 21/40 staging/lustre/mdt: HSM coordinator agent interface
In reply to Peng TaoFrom: JC Lafoucriere <[email protected]>
To move data with external storage, HSM coordinator uses a Copy Tool running on a client named agent. This patch implements the interface for these agents.
Lustre-change: http://review.whamcloud.com/6534
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3342
--- .../lustre/lustre/include/lustre/lustre_user.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_user.h +3 -3
@@ -428,8 +428,8 @@ struct obd_uuid {char uuid[UUID_MAX ]; }; -static inline int obd_uuid_equals (const struct obd_uuid *u1, -const struct obd_uuid *u2) +static inline bool obd_uuid_equals (const struct obd_uuid *u1, +const struct obd_uuid *u2) {return strcmp ((char *)u1->uuid, (char *)u2->uuid) ==0 ; }
@@ -446,7 +446,7 @@ static inline void obd_str2uuid(struct obd_uuid *uuid, const char *tmp) }/* For printf's only, make sure uuid is terminated */ -static inline char *obd_uuid2str (struct obd_uuid *uuid) +static inline char *obd_uuid2str (const struct obd_uuid *uuid) {if (uuid->uuid[sizeof (*uuid) -1 ] !='\0' ) {/* Obviously not safe, but for printfs, no real harm done...
Signature · 1 line
1.7.9.5
PATCH 26/40 staging/lustre/build: fix compilation issue with is_compat_task
In reply to Peng TaoFrom: Dmitry Eremin <[email protected]>
After removing LIBCFS_HAVE_IS_COMPAT_TASK test we have a compilation issue with kernels configured without CONFIG_COMPAT.
Lustre-change: http://review.whamcloud.com/7118
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2800
--- .../staging/lustre/lustre/llite/llite_internal.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/staging/lustre/lustre/llite/llite_internal.h +6 -1
@@ -650,7 +650,12 @@ static inline int ll_need_32bit_api(struct ll_sb_info *sbi)#if BITS_PER_LONG ==32 return 1 ;#else -return unlikely (is_compat_task () || (sbi->ll_flags &LL_SBI_32BIT_API )); + return unlikely( +#ifdef CONFIG_COMPAT +is_compat_task () || +#endif + (sbi->ll_flags &LL_SBI_32BIT_API ) + ); #endif }
Signature · 1 line
1.7.9.5
PATCH 27/40 staging/lustre/ptlrpc: Fix a crash when dereferencing NULL pointer
In reply to Peng TaoFrom: Amir Shehata <[email protected]>
When a system runs out of memory and the function ptlrpc_register_bulk() is called from ptl_send_rpc() the call to LNetMEAttach() fails due to failure to allocate memory. This forces the code into an error path, which most probably previously went untested. The error path:
if (rc != 0) {
CERROR("%s: LNetMEAttach failed x"LPU64"/%d: rc = %dn",
desc->bd_export->exp_obd->obd_name, xid,
posted_md, rc);
break;
}This print assumes that desc->bd_export is not NULL. However, it is. In fact it is expected to be NULL. desc->bd_import is the correct structure to access in this case.
Lustre-change: http://review.whamcloud.com/7121
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3585
--- drivers/staging/lustre/lustre/ptlrpc/niobuf.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/staging/lustre/lustre/ptlrpc/niobuf.c +3 -3
@@ -179,7 +179,7 @@ int ptlrpc_register_bulk(struct ptlrpc_request *req) LNET_UNLINK, LNET_INS_AFTER, &me_h);if (rc !=0 ) {CERROR ("%s: LNetMEAttach failed x" LPU64 "/%d: rc = %d\n" , - desc->bd_export->exp_obd->obd_name, xid, + desc->bd_import->imp_obd->obd_name, xid, posted_md, rc);break ; }
@@ -189,7 +189,7 @@ int ptlrpc_register_bulk(struct ptlrpc_request *req) &desc->bd_mds[posted_md]);if (rc !=0 ) {CERROR ("%s: LNetMDAttach failed x" LPU64 "/%d: rc = %d\n" , - desc->bd_export->exp_obd->obd_name, xid, + desc->bd_import->imp_obd->obd_name, xid, posted_md, rc); rc2 =LNetMEUnlink (me_h);LASSERT (rc2 ==0 );
@@ -219,7 +219,7 @@ int ptlrpc_register_bulk(struct ptlrpc_request *req)/* Holler if peer manages to touch buffers before he knows the xid */ if (desc->bd_md_count != total_md)CWARN ("%s: Peer %s touched %d buffers while I registered\n" , - desc->bd_export->exp_obd->obd_name,libcfs_id2str (peer), + desc->bd_import->imp_obd->obd_name,libcfs_id2str (peer), total_md - desc->bd_md_count);spin_unlock (&desc->bd_lock);
Signature · 1 line
1.7.9.5
PATCH 28/40 staging/lustre/hsm: Add hsm_release feature.
In reply to Peng TaoFrom: Jinshan Xiong <[email protected]>
HSM Release is one of the key feature of HSM. To perform HSM release, clients need to acquire the file lease exclusivelt and flush dirty cache from clients. A special close REQ will be sent to the MDT to release the lease and get rid of OST objects.
Lustre-change: http://review.whamcloud.com/7028
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-1333
--- .../lustre/lustre/include/lustre/lustre_idl.h | 14 +++- .../lustre/lustre/include/lustre/lustre_user.h | 11 ++- .../lustre/lustre/include/lustre_req_layout.h | 2 + drivers/staging/lustre/lustre/include/obd.h | 6 +- .../staging/lustre/lustre/lclient/lcommon_misc.c | 4 +- drivers/staging/lustre/lustre/llite/dir.c | 24 +++++- drivers/staging/lustre/lustre/llite/file.c | 87 +++++++++++++++++--- .../staging/lustre/lustre/llite/llite_internal.h | 3 +- drivers/staging/lustre/lustre/llite/vvp_object.c | 2 +- .../staging/lustre/lustre/lov/lov_cl_internal.h | 16 ++++ drivers/staging/lustre/lustre/lov/lov_object.c | 35 +++++--- drivers/staging/lustre/lustre/mdc/mdc_lib.c | 26 +++++- drivers/staging/lustre/lustre/mdc/mdc_request.c | 23 +++++- drivers/staging/lustre/lustre/ptlrpc/layout.c | 19 +++++ .../staging/lustre/lustre/ptlrpc/pack_generic.c | 7 ++ 15 files changed, 242 insertions(+), 37 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +13 -1
@@ -1750,6 +1750,7 @@ static inline __u32 lov_mds_md_size(__u16 stripes, __u32 lmm_magic)#define OBD_MD_FLRMTRGETFACL (0x0008000000000000ULL)/* lfs rgetfacl case */ #define OBD_MD_FLDATAVERSION (0x0010000000000000ULL)/* iversion sum */ +#define OBD_MD_FLRELEASED (0x0020000000000000ULL)/* file released */ #define OBD_MD_FLGETATTR (OBD_MD_FLID | OBD_MD_FLATIME | OBD_MD_FLMTIME | \ OBD_MD_FLCTIME | OBD_MD_FLSIZE | OBD_MD_FLBLKSZ | \
@@ -2386,6 +2387,7 @@ extern void lustre_swab_mdt_rec_setattr (struct mdt_rec_setattr *sa);* delegation, succeed if it's not * being opened with conflict mode. */ +#define MDS_OPEN_RELEASE 02000000000000ULL/* Open the file for HSM release */ /* permission for create non-directory file */ #define MAY_CREATE (1 << 7)
@@ -2404,7 +2406,7 @@ extern void lustre_swab_mdt_rec_setattr (struct mdt_rec_setattr *sa);/* lfs rgetfacl permission check */ #define MAY_RGETFACL (1 << 14) -enum { +enum mds_op_bias {MDS_CHECK_SPLIT =1 <<0 ,MDS_CROSS_REF =1 <<1 ,MDS_VTX_BYPASS =1 <<2 ,
@@ -2417,6 +2419,7 @@ enum {MDS_DATA_MODIFIED =1 <<9 ,MDS_CREATE_VOLATILE =1 <<10 ,MDS_OWNEROVERRIDE =1 <<11 , + MDS_HSM_RELEASE = 1 << 12, };/* instance of mdt_reint_rec */
@@ -3740,5 +3743,14 @@ struct mdc_swap_layouts {void lustre_swab_swap_layouts (struct mdc_swap_layouts *msl); +struct close_data { +struct lustre_handle cd_handle; +struct lu_fid cd_fid; +__u64 cd_data_version; +__u64 cd_reserved[8 ]; +}; + +void lustre_swab_close_data (struct close_data *data); +#endif /** @} lustreidl */
drivers/staging/lustre/lustre/include/lustre/lustre_user.h +7 -4
@@ -623,10 +623,13 @@ struct if_quotactl { };/* swap layout flags */ -#define SWAP_LAYOUTS_CHECK_DV1 (1 << 0) -#define SWAP_LAYOUTS_CHECK_DV2 (1 << 1) -#define SWAP_LAYOUTS_KEEP_MTIME (1 << 2) -#define SWAP_LAYOUTS_KEEP_ATIME (1 << 3) +#define SWAP_LAYOUTS_CHECK_DV1 (1 << 0) +#define SWAP_LAYOUTS_CHECK_DV2 (1 << 1) +#define SWAP_LAYOUTS_KEEP_MTIME (1 << 2) +#define SWAP_LAYOUTS_KEEP_ATIME (1 << 3) + +/* Swap XATTR_NAME_HSM as well, only on the MDT so far */ +#define SWAP_LAYOUTS_MDS_HSM (1 << 31)struct lustre_swap_layouts {__u64 sl_flags;__u32 sl_fd;
drivers/staging/lustre/lustre/include/lustre_req_layout.h +2 -0
@@ -164,6 +164,7 @@ extern struct req_format RQF_UPDATE_OBJ;*/ extern struct req_format RQF_MDS_GETATTR_NAME ;extern struct req_format RQF_MDS_CLOSE ; +extern struct req_format RQF_MDS_RELEASE_CLOSE ;extern struct req_format RQF_MDS_PIN ;extern struct req_format RQF_MDS_UNPIN ;extern struct req_format RQF_MDS_CONNECT ;
@@ -262,6 +263,7 @@ extern struct req_msg_field RMF_GETINFO_VAL;extern struct req_msg_field RMF_GETINFO_VALLEN ;extern struct req_msg_field RMF_GETINFO_KEY ;extern struct req_msg_field RMF_IDX_INFO ; +extern struct req_msg_field RMF_CLOSE_DATA ;/* * connection handle received in MDS_CONNECT request.
drivers/staging/lustre/lustre/include/obd.h +5 -1
@@ -1070,7 +1070,7 @@ struct md_op_data {struct obd_capa *op_capa2;/* Various operation flags. */ -__u32 op_bias; +enum mds_op_bias op_bias;/* Operation type */ __u32 op_opc;
@@ -1084,6 +1084,10 @@ struct md_op_data {/* used to transfer info between the stacks of MD client * see enum op_cli_flags */ __u32 op_cli_flags; + +/* File object data version for HSM release, on client */ +__u64 op_data_version; +struct lustre_handle op_lease_handle; };enum op_cli_flags {
drivers/staging/lustre/lustre/lclient/lcommon_misc.c +3 -1
@@ -140,7 +140,9 @@ int cl_get_grouplock(struct cl_object *obj, unsigned long gid, int nonblock, rc =cl_io_init (env, io,CIT_MISC , io->ci_obj);if (rc) { -LASSERT (rc <0 ); +/* Does not make sense to take GL for released layout */ +if (rc >0 ) + rc = -ENOTSUPP ;cl_env_put (env, &refcheck);return rc; }
drivers/staging/lustre/lustre/llite/dir.c +22 -2
@@ -1809,8 +1809,28 @@ out_rmdir:return -EFAULT ; } - rc =obd_iocontrol (cmd,ll_i2mdexp (inode), totalsize, - hur,NULL ); +if (hur->hur_request.hr_action ==HUA_RELEASE ) { +const struct lu_fid *fid; +struct inode *f; +int i; + +for (i =0 ; i < hur->hur_request.hr_itemcount; i++) { + fid = &hur->hur_user_item[i].hui_fid; + f =search_inode_for_lustre (inode->i_sb, fid); +if (IS_ERR (f)) { + rc =PTR_ERR (f); +break ; + } + + rc =ll_hsm_release (f); +iput (f); +if (rc !=0 ) +break ; + } + }else { + rc =obd_iocontrol (cmd,ll_i2mdexp (inode), totalsize, + hur,NULL ); + }OBD_FREE_LARGE (hur, totalsize);
drivers/staging/lustre/lustre/llite/file.c +75 -12
165 lines not on this page · raw message
drivers/staging/lustre/lustre/llite/llite_internal.h +2 -1
@@ -787,9 +787,10 @@ int ll_get_grouplock(struct inode *inode, struct file *file, unsigned long arg);int ll_put_grouplock (struct inode *inode,struct file *file,unsigned long arg);int ll_fid2path (struct inode *inode,void *arg);int ll_data_version (struct inode *inode,__u64 *data_version,int extent_lock); +int ll_hsm_release (struct inode *inode);struct obd_client_handle *ll_lease_open (struct inode *inode,struct file *file, -fmode_t mode); +fmode_t mode,__u64 flags);int ll_lease_close (struct obd_client_handle *och,struct inode *inode,bool *lease_broken);
drivers/staging/lustre/lustre/llite/vvp_object.c +1 -1
@@ -138,7 +138,7 @@ int vvp_conf_set(const struct lu_env *env, struct cl_object *obj, lli->lli_layout_gen, conf->u.coc_md->lsm->lsm_layout_gen); - lli->lli_has_smd = true; + lli->lli_has_smd =lsm_has_objects (conf->u.coc_md->lsm); lli->lli_layout_gen = conf->u.coc_md->lsm->lsm_layout_gen; }else { CDEBUG(D_VFSTRACE, "layout lock destroyed: %u.\n",
drivers/staging/lustre/lustre/lov/lov_cl_internal.h +16 -0
@@ -168,6 +168,22 @@ enum lov_layout_type { LLT_NR }; +static inline char *llt2str (enum lov_layout_type llt) +{ +switch (llt) { +case LLT_EMPTY : +return "EMPTY" ; +case LLT_RAID0 : +return "RAID0" ; +case LLT_RELEASED : +return "RELEASED" ; +case LLT_NR : +LBUG (); + } +LBUG (); +return "" ; +} +/** * lov-specific file state. *
drivers/staging/lustre/lustre/lov/lov_object.c +25 -10
@@ -393,13 +393,13 @@ static int lov_print_empty(const struct lu_env *env, void *cookie,static int lov_print_raid0 (const struct lu_env *env,void *cookie,lu_printer_t p,const struct lu_object *o) { -struct lov_object *lov =lu2lov (o); -struct lov_layout_raid0 *r0 =lov_r0 (lov); -struct lov_stripe_md *lsm = lov->lo_lsm; -int i; +struct lov_object *lov =lu2lov (o); +struct lov_layout_raid0 *r0 =lov_r0 (lov); +struct lov_stripe_md *lsm = lov->lo_lsm; +int i; - (*p)(env, cookie,"stripes: %d, %svalid, lsm{%p 0x%08X %d %u %u}: \n" , - r0->lo_nr, lov->lo_layout_invalid ?"in" :"" , lsm, + (*p)(env, cookie,"stripes: %d, %s, lsm{%p 0x%08X %d %u %u}:\n" , + r0->lo_nr, lov->lo_layout_invalid ?"invalid" :"valid" , lsm, lsm->lsm_magic,atomic_read (&lsm->lsm_refc), lsm->lsm_stripe_count, lsm->lsm_layout_gen);for (i =0 ; i < r0->lo_nr; ++i) {
@@ -408,8 +408,9 @@ static int lov_print_raid0(const struct lu_env *env, void *cookie,if (r0->lo_sub[i] !=NULL ) { sub =lovsub2lu (r0->lo_sub[i]);lu_object_print (env, cookie, p, sub); - }else + }else { (*p)(env, cookie,"sub %d absent\n" , i); + } }return 0 ; }
@@ -417,7 +418,14 @@ static int lov_print_raid0(const struct lu_env *env, void *cookie,static int lov_print_released (const struct lu_env *env,void *cookie,lu_printer_t p,const struct lu_object *o) { - (*p)(env, cookie,"released\n" ); +struct lov_object *lov =lu2lov (o); +struct lov_stripe_md *lsm = lov->lo_lsm; + + (*p)(env, cookie, +"released: %s, lsm{%p 0x%08X %d %u %u}:\n" , + lov->lo_layout_invalid ?"invalid" :"valid" , lsm, + lsm->lsm_magic,atomic_read (&lsm->lsm_refc), + lsm->lsm_stripe_count, lsm->lsm_layout_gen);return 0 ; }
@@ -662,6 +670,10 @@ static int lov_layout_change(const struct lu_env *unused,return PTR_ERR (env); } +CDEBUG (D_INODE ,DFID " from %s to %s\n" , +PFID (lu_object_fid (lov2lu (lov))), +llt2str (lov->lo_type),llt2str (llt)); + old_ops = &lov_dispatch[lov->lo_type]; new_ops = &lov_dispatch[llt];
@@ -750,8 +762,9 @@ static int lov_conf_set(const struct lu_env *env, struct cl_object *obj,if (conf->u.coc_md !=NULL ) lsm = conf->u.coc_md->lsm;if ((lsm ==NULL && lov->lo_lsm ==NULL ) || - (lsm !=NULL && lov->lo_lsm !=NULL && - lov->lo_lsm->lsm_layout_gen == lsm->lsm_layout_gen)) { + ((lsm !=NULL && lov->lo_lsm !=NULL ) && + (lov->lo_lsm->lsm_layout_gen == lsm->lsm_layout_gen) && + (lov->lo_lsm->lsm_pattern == lsm->lsm_pattern))) {/* same version of layout */ lov->lo_layout_invalid = false;GOTO (out, result =0 );
@@ -767,6 +780,8 @@ static int lov_conf_set(const struct lu_env *env, struct cl_object *obj, out:lov_conf_unlock (lov); +CDEBUG (D_INODE ,DFID " lo_layout_invalid=%d\n" , +PFID (lu_object_fid (lov2lu (lov))), lov->lo_layout_invalid);return result; }
drivers/staging/lustre/lustre/mdc/mdc_lib.c +25 -1
@@ -179,7 +179,8 @@ static __u64 mds_pack_open_flags(__u64 flags, __u32 mode)__u64 cr_flags = (flags & (FMODE_READ |FMODE_WRITE |MDS_OPEN_HAS_EA |MDS_OPEN_HAS_OBJS |MDS_OPEN_OWNEROVERRIDE |MDS_OPEN_LOCK | -MDS_OPEN_BY_FID |MDS_OPEN_LEASE )); +MDS_OPEN_BY_FID |MDS_OPEN_LEASE | +MDS_OPEN_RELEASE ));if (flags &O_CREAT ) cr_flags |=MDS_OPEN_CREAT ;if (flags &O_EXCL )
@@ -490,6 +491,28 @@ void mdc_getattr_pack(struct ptlrpc_request *req, __u64 valid, int flags, } } +static void mdc_hsm_release_pack (struct ptlrpc_request *req, +struct md_op_data *op_data) +{ +if (op_data->op_bias &MDS_HSM_RELEASE ) { +struct close_data *data; +struct ldlm_lock *lock; + + data =req_capsule_client_get (&req->rq_pill, &RMF_CLOSE_DATA ); +LASSERT (data !=NULL ); + + lock =ldlm_handle2lock (&op_data->op_lease_handle); +if (lock !=NULL ) { + data->cd_handle = lock->l_remote_handle; +ldlm_lock_put (lock); + } +ldlm_cli_cancel (&op_data->op_lease_handle,LCF_LOCAL ); + + data->cd_data_version = op_data->op_data_version; + data->cd_fid = op_data->op_fid2; + } +} +void mdc_close_pack (struct ptlrpc_request *req,struct md_op_data *op_data) {struct mdt_ioepoch *epoch;
@@ -501,6 +524,7 @@ void mdc_close_pack(struct ptlrpc_request *req, struct md_op_data *op_data)mdc_setattr_pack_rec (rec, op_data);mdc_pack_capa (req, &RMF_CAPA1 , op_data->op_capa1);mdc_ioepoch_pack (epoch, op_data); +mdc_hsm_release_pack (req, op_data); }static int mdc_req_avail (struct client_obd *cli,struct mdc_cache_waiter *mcw)
drivers/staging/lustre/lustre/mdc/mdc_request.c +20 -3
@@ -800,10 +800,27 @@ int mdc_close(struct obd_export *exp, struct md_op_data *op_data, {struct obd_device *obd =class_exp2obd (exp);struct ptlrpc_request *req; -int rc; +struct req_format *req_fmt; +int rc; +int saved_rc =0 ; + + + req_fmt = &RQF_MDS_CLOSE ; +if (op_data->op_bias &MDS_HSM_RELEASE ) { + req_fmt = &RQF_MDS_RELEASE_CLOSE ; + +/* allocate a FID for volatile file */ + rc =mdc_fid_alloc (exp, &op_data->op_fid2, op_data); +if (rc <0 ) { +CERROR ("%s: " DFID " failed to allocate FID: %d\n" , + obd->obd_name,PFID (&op_data->op_fid1), rc); +/* save the errcode and proceed to close */ + saved_rc = rc; + } + } *request =NULL ; - req =ptlrpc_request_alloc (class_exp2cliimp (exp), &RQF_MDS_CLOSE ); + req =ptlrpc_request_alloc (class_exp2cliimp (exp), req_fmt);if (req ==NULL )return -ENOMEM ;
@@ -893,7 +910,7 @@ int mdc_close(struct obd_export *exp, struct md_op_data *op_data, } *request = req;mdc_close_handle_reply (req, op_data, rc); -return rc; +return rc <0 ? rc : saved_rc; } int mdc_done_writing(struct obd_export *exp, struct md_op_data *op_data,
drivers/staging/lustre/lustre/ptlrpc/layout.c +19 -0
@@ -145,6 +145,14 @@ static const struct req_msg_field *mdt_close_client[] = { &RMF_CAPA1 }; +static const struct req_msg_field *mdt_release_close_client[] = { + &RMF_PTLRPC_BODY , + &RMF_MDT_EPOCH , + &RMF_REC_REINT , + &RMF_CAPA1 , + &RMF_CLOSE_DATA +}; +static const struct req_msg_field *obd_statfs_server[] = { &RMF_PTLRPC_BODY , &RMF_OBD_STATFS
@@ -666,6 +674,7 @@ static struct req_format *req_formats[] = { &RQF_MDS_GETXATTR , &RQF_MDS_SYNC , &RQF_MDS_CLOSE , + &RQF_MDS_RELEASE_CLOSE , &RQF_MDS_PIN , &RQF_MDS_UNPIN , &RQF_MDS_READPAGE,
@@ -885,6 +894,11 @@ struct req_msg_field RMF_PTLRPC_BODY = sizeof(struct ptlrpc_body), lustre_swab_ptlrpc_body, NULL);EXPORT_SYMBOL (RMF_PTLRPC_BODY ); +struct req_msg_field RMF_CLOSE_DATA = +DEFINE_MSGF ("data_version" ,0 , +sizeof (struct close_data ), lustre_swab_close_data,NULL ); +EXPORT_SYMBOL (RMF_CLOSE_DATA ); +struct req_msg_field RMF_OBD_STATFS =DEFINE_MSGF ("obd_statfs" ,0 ,sizeof (struct obd_statfs ), lustre_swab_obd_statfs,NULL );
@@ -1412,6 +1426,11 @@ struct req_format RQF_MDS_CLOSE = mdt_close_client, mds_last_unlink_server);EXPORT_SYMBOL (RQF_MDS_CLOSE ); +struct req_format RQF_MDS_RELEASE_CLOSE = +DEFINE_REQ_FMT0 ("MDS_CLOSE" , + mdt_release_close_client, mds_last_unlink_server); +EXPORT_SYMBOL (RQF_MDS_RELEASE_CLOSE ); +struct req_format RQF_MDS_PIN =DEFINE_REQ_FMT0 ("MDS_PIN" , mdt_body_capa, mdt_body_only);
drivers/staging/lustre/lustre/ptlrpc/pack_generic.c +7 -0
@@ -2565,3 +2565,10 @@ void lustre_swab_swap_layouts(struct mdc_swap_layouts *msl)__swab64s (&msl->msl_flags); }EXPORT_SYMBOL (lustre_swab_swap_layouts); + +void lustre_swab_close_data (struct close_data *cd) +{ +lustre_swab_lu_fid (&cd->cd_fid); +__swab64s (&cd->cd_data_version); +} +EXPORT_SYMBOL (lustre_swab_close_data);
Signature · 1 line
1.7.9.5
PATCH 30/40 staging/lustre/xattr: separate ACL and XATTR caches
In reply to Peng TaoFrom: Nathaniel Clark <[email protected]>
This patch separates ACL and XATTR caches, so that when updating an ACL only LOOKUP lock is needed and when updating another XATTR only XATTR lock is needed.
This patch also reverts XATTR cache support for setxattr because client performing REINT under even PR lock can deadlock if an active server operation (like unlink) attempts to cancel all locks, and setxattr has to wait for it (MDC max-in-flight is 1).
This patch also drops mot_xattr_sem because there is no use case for FLXATTRLS with locking.
This patch also takes into account that osd_xattr_list may be changing lu_buf, so it does not assume that the buf is unchanged after the call.
This patch disables the r/o cache if the data is unreasonably large (larger than maximum single EA size).
Lustre-change: http://review.whamcloud.com/7208
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3669
--- .../lustre/lustre/include/lustre/lustre_idl.h | 1 - drivers/staging/lustre/lustre/ldlm/ldlm_lock.c | 2 - .../staging/lustre/lustre/llite/llite_internal.h | 7 -- drivers/staging/lustre/lustre/llite/xattr.c | 39 ++++--- drivers/staging/lustre/lustre/llite/xattr_cache.c | 119 ++++---------------- drivers/staging/lustre/lustre/mdc/mdc_internal.h | 2 +- drivers/staging/lustre/lustre/mdc/mdc_locks.c | 9 +- drivers/staging/lustre/lustre/mdc/mdc_reint.c | 2 +- drivers/staging/lustre/lustre/mdc/mdc_request.c | 30 ++++- drivers/staging/lustre/lustre/ptlrpc/layout.c | 3 +- 10 files changed, 74 insertions(+), 140 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +0 -1
@@ -1760,7 +1760,6 @@ static inline __u32 lov_mds_md_size(__u16 stripes, __u32 lmm_magic) OBD_MD_FLGID | OBD_MD_FLFLAGS | OBD_MD_FLNLINK | \ OBD_MD_FLGENER | OBD_MD_FLRDEV | OBD_MD_FLGROUP) -#define OBD_MD_FLXATTRLOCKED OBD_MD_FLGETATTRLOCK#define OBD_MD_FLXATTRALL (OBD_MD_FLXATTR | OBD_MD_FLXATTRLS)/* don't forget obdo_fid which is way down at the bottom so it can
drivers/staging/lustre/lustre/ldlm/ldlm_lock.c +0 -2
@@ -145,8 +145,6 @@ char *ldlm_it2str(int it)return "getxattr" ;case IT_LAYOUT :return "layout" ; -case IT_SETXATTR : -return "setxattr" ;default :CERROR ("Unknown intent %d\n" , it);return "UNKNOWN" ;
drivers/staging/lustre/lustre/llite/llite_internal.h +0 -7
@@ -295,13 +295,6 @@ int ll_xattr_cache_get(struct inode *inode,size_t size, __u64 valid); -int ll_xattr_cache_update (struct inode *inode, -const char *name, -const char *newval, -size_t size, -__u64 valid, -int flags); -/* * Locking to guarantee consistency of non-atomic updates to long long i_size, * consistency between file size and KMS.
drivers/staging/lustre/lustre/llite/xattr.c +24 -15
@@ -183,17 +183,11 @@ int ll_setxattr_common(struct inode *inode, const char *name, valid |= rce_ops2valid(rce->rce_ops); }#endif -if (sbi->ll_xattr_cache_enabled && - (rce ==NULL || rce->rce_ops ==RMT_LSETFACL )) { - rc =ll_xattr_cache_update (inode, name, pv, size, valid, flags); - }else { - oc =ll_mdscapa_get (inode); - rc =md_setxattr (sbi->ll_md_exp,ll_inode2fid (inode), oc, - valid, name, pv, size,0 , flags, -ll_i2suppgid (inode), &req); -capa_put (oc); - } - + oc =ll_mdscapa_get (inode); + rc =md_setxattr (sbi->ll_md_exp,ll_inode2fid (inode), oc, + valid, name, pv, size,0 , flags, +ll_i2suppgid (inode), &req); +capa_put (oc);#ifdef CONFIG_FS_POSIX_ACL if (new_value !=NULL )lustre_posix_acl_xattr_free (new_value, size);
@@ -292,6 +286,7 @@ int ll_getxattr_common(struct inode *inode, const char *name,void *xdata;struct obd_capa *oc;struct rmtacl_ctl_entry *rce =NULL ; +struct ll_inode_info *lli =ll_i2info (inode);CDEBUG (D_VFSTRACE ,"VFS Op:inode=%lu/%u(%p)\n" , inode->i_ino, inode->i_generation, inode);
@@ -339,7 +334,7 @@ int ll_getxattr_common(struct inode *inode, const char *name,*/ if (xattr_type ==XATTR_ACL_ACCESS_T && !(sbi->ll_flags &LL_SBI_RMT_CLIENT )) { -struct ll_inode_info *lli =ll_i2info (inode); +struct posix_acl *acl;spin_lock (&lli->lli_lock);
@@ -358,13 +353,27 @@ int ll_getxattr_common(struct inode *inode, const char *name,#endif do_getxattr: -if (sbi->ll_xattr_cache_enabled && (rce ==NULL || - rce->rce_ops ==RMT_LGETFACL || - rce->rce_ops ==RMT_LSETFACL )) { +if (sbi->ll_xattr_cache_enabled && xattr_type !=XATTR_ACL_ACCESS_T ) { rc =ll_xattr_cache_get (inode, name, buffer, size, valid); +if (rc == -EAGAIN ) + goto getxattr_nocache;if (rc <0 )GOTO (out_xattr, rc); + +/* Add "system.posix_acl_access" to the list */ +if (lli->lli_posix_acl !=NULL && valid &OBD_MD_FLXATTRLS ) { +if (size ==0 ) { + rc +=sizeof (XATTR_NAME_ACL_ACCESS ); + }else if (size - rc >=sizeof (XATTR_NAME_ACL_ACCESS )) { +memcpy (buffer + rc,XATTR_NAME_ACL_ACCESS , +sizeof (XATTR_NAME_ACL_ACCESS )); + rc +=sizeof (XATTR_NAME_ACL_ACCESS ); + }else { +GOTO (out_xattr, rc = -ERANGE ); + } + } }else { +getxattr_nocache: oc =ll_mdscapa_get (inode); rc = md_getxattr(sbi->ll_md_exp, ll_inode2fid(inode), oc, valid | (rce ? rce_ops2valid(rce->rce_ops) : 0),
drivers/staging/lustre/lustre/llite/xattr_cache.c +19 -100
175 lines not on this page · raw message
drivers/staging/lustre/lustre/mdc/mdc_internal.h +1 -1
@@ -101,7 +101,7 @@ int mdc_enqueue(struct obd_export *exp, struct ldlm_enqueue_info *einfo, struct lustre_handle *lockh, void *lmm, int lmmsize, struct ptlrpc_request **req, __u64 extra_lock_flags); -int mdc_resource_get_unused (struct obd_export *exp,struct lu_fid *fid, +int mdc_resource_get_unused (struct obd_export *exp,const struct lu_fid *fid,struct list_head *cancels,ldlm_mode_t mode,__u64 bits);/* mdc/mdc_request.c */
drivers/staging/lustre/lustre/mdc/mdc_locks.c +1 -8
@@ -379,13 +379,6 @@ mdc_intent_getxattr_pack(struct obd_export *exp,mdc_set_capa_size (req, &RMF_CAPA1 , op_data->op_capa1); -if (it->it_op ==IT_SETXATTR ) -/* If we want to upgrade to LCK_PW, let's cancel LCK_PR -* locks now. This avoids unnecessary ASTs. */ - count =mdc_resource_get_unused (exp, &op_data->op_fid1, - &cancels,LCK_PW , -MDS_INODELOCK_XATTR ); - rc =ldlm_prep_enqueue_req (exp, req, &cancels, count);if (rc) {ptlrpc_request_free (req);
@@ -843,7 +836,7 @@ resend:return -EOPNOTSUPP ; req =mdc_intent_layout_pack (exp, it, op_data); lvb_type =LVB_T_LAYOUT ; - }else if (it->it_op & (IT_GETXATTR |IT_SETXATTR )) { + }else if (it->it_op &IT_GETXATTR ) { req =mdc_intent_getxattr_pack (exp, it, op_data); }else {LBUG ();
drivers/staging/lustre/lustre/mdc/mdc_reint.c +1 -1
@@ -66,7 +66,7 @@ static int mdc_reint(struct ptlrpc_request *request,/* Find and cancel locally locks matched by inode @bits & @mode in the resource * found by @fid. Found locks are added into @cancel list. Returns the amount of * locks added to @cancels list. */ -int mdc_resource_get_unused (struct obd_export *exp,struct lu_fid *fid, +int mdc_resource_get_unused (struct obd_export *exp,const struct lu_fid *fid,struct list_head *cancels,ldlm_mode_t mode,__u64 bits) {
drivers/staging/lustre/lustre/mdc/mdc_request.c +26 -4
@@ -355,10 +355,32 @@ static int mdc_xattr_common(struct obd_export *exp,const struct req_format *fmt, input_size); } - rc =ptlrpc_request_pack (req,LUSTRE_MDS_VERSION , opcode); -if (rc) { -ptlrpc_request_free (req); -return rc; +/* Flush local XATTR locks to get rid of a possible cancel RPC */ +if (opcode ==MDS_REINT &&fid_is_sane (fid) && + exp->exp_connect_data.ocd_ibits_known &MDS_INODELOCK_XATTR ) { +LIST_HEAD (cancels); +int count; + +/* Without that packing would fail */ +if (input_size ==0 ) +req_capsule_set_size (&req->rq_pill, &RMF_EADATA , +RCL_CLIENT ,0 ); + + count =mdc_resource_get_unused (exp, fid, + &cancels,LCK_EX , +MDS_INODELOCK_XATTR ); + + rc =mdc_prep_elc_req (exp, req,MDS_REINT , &cancels, count); +if (rc) { +ptlrpc_request_free (req); +return rc; + } + }else { + rc =ptlrpc_request_pack (req,LUSTRE_MDS_VERSION , opcode); +if (rc) { +ptlrpc_request_free (req); +return rc; + } }if (opcode ==MDS_REINT ) {
drivers/staging/lustre/lustre/ptlrpc/layout.c +2 -1
@@ -295,7 +295,8 @@ static const struct req_msg_field *mds_reint_setxattr_client[] = { &RMF_REC_REINT , &RMF_CAPA1 , &RMF_NAME , - &RMF_EADATA + &RMF_EADATA , + &RMF_DLM_REQ }; static const struct req_msg_field *mdt_swap_layouts[] = {
Signature · 1 line
1.7.9.5
PATCH 32/40 staging/lustre/api: HSM import uses new released pattern
In reply to Peng TaoFrom: JC Lafoucriere <[email protected]>
Import creates a released file using new RAID pattern flag Import used a new ioctl() to implement the import in the client kernel.
Add a -L|--layout option to lfs getstripe and lfs find
Lustre-change: http://review.whamcloud.com/6536
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3363
--- .../lustre/lustre/include/lustre/lustre_user.h | 17 +- .../lustre/lustre/include/lustre/lustreapi.h | 172 +++++++++++--------- drivers/staging/lustre/lustre/llite/file.c | 118 +++++++++++--- .../staging/lustre/lustre/llite/llite_internal.h | 2 +- drivers/staging/lustre/lustre/llite/llite_lib.c | 21 ++- drivers/staging/lustre/lustre/ptlrpc/wiretest.c | 40 +++++ 6 files changed, 261 insertions(+), 109 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_user.h +16 -1
@@ -244,9 +244,9 @@ struct ost_id {#define LL_IOC_LMV_SETSTRIPE _IOWR('f', 240, struct lmv_user_md)#define LL_IOC_LMV_GETSTRIPE _IOWR('f', 241, struct lmv_user_md)#define LL_IOC_REMOVE_ENTRY _IOWR('f', 242, __u64) -#define LL_IOC_SET_LEASE _IOWR('f', 243, long)#define LL_IOC_GET_LEASE _IO('f', 244) +#define LL_IOC_HSM_IMPORT _IOWR('f', 245, struct hsm_user_import)#define LL_STATFS_LMV 1#define LL_STATFS_LOV 2
@@ -1130,6 +1130,21 @@ static inline int hal_size(struct hsm_action_list *hal)return sz; } +/* HSM file import +* describe the attributes to be set on imported file +*/ +struct hsm_user_import { +__u64 hui_size; +__u64 hui_atime; +__u64 hui_mtime; +__u32 hui_atime_ns; +__u32 hui_mtime_ns; +__u32 hui_uid; +__u32 hui_gid; +__u32 hui_mode; +__u32 hui_archive_id; +}; +/* Copytool progress reporting */ #define HP_FLAG_COMPLETED 0x01#define HP_FLAG_RETRY 0x02
drivers/staging/lustre/lustre/include/lustre/lustreapi.h +94 -78
205 lines not on this page · raw message
drivers/staging/lustre/lustre/llite/file.c +98 -20
148 lines not on this page · raw message
drivers/staging/lustre/lustre/llite/llite_internal.h +1 -1
@@ -839,7 +839,7 @@ void ll_kill_super(struct super_block *sb);struct inode *ll_inode_from_resource_lock (struct ldlm_lock *lock);struct inode *ll_inode_from_lock (struct ldlm_lock *lock);void ll_clear_inode (struct inode *inode); -int ll_setattr_raw (struct dentry *dentry,struct iattr *attr); +int ll_setattr_raw (struct dentry *dentry,struct iattr *attr,bool hsm_import);int ll_setattr (struct dentry *de,struct iattr *attr);int ll_statfs (struct dentry *de,struct kstatfs *sfs); int ll_statfs_internal(struct super_block *sb, struct obd_statfs *osfs,
drivers/staging/lustre/lustre/llite/llite_lib.c +12 -9
@@ -1362,8 +1362,10 @@ static int ll_setattr_ost(struct inode *inode, struct iattr *attr)* to the OST with the punch RPC, otherwise we do an explicit setattr RPC. * I don't believe it is possible to get e.g. ATTR_MTIME_SET and ATTR_SIZE * at the same time. +* +* In case of HSMimport, we only set attr on MDS. */ -int ll_setattr_raw (struct dentry *dentry,struct iattr *attr) +int ll_setattr_raw (struct dentry *dentry,struct iattr *attr,bool hsm_import) {struct inode *inode = dentry->d_inode;struct ll_inode_info *lli =ll_i2info (inode);
@@ -1373,9 +1375,10 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr)int rc =0 , rc1 =0 ;CDEBUG (D_VFSTRACE ,"%s: setattr inode %p/fid:" DFID " from %llu to %llu, " -"valid %x\n" ,ll_get_fsname (inode->i_sb,NULL ,0 ), inode, +"valid %x, hsm_import %d\n" , +ll_get_fsname (inode->i_sb,NULL ,0 ), inode,PFID (&lli->lli_fid),i_size_read (inode), attr->ia_size, - attr->ia_valid); + attr->ia_valid, hsm_import);if (attr->ia_valid &ATTR_SIZE ) {/* Check new size against VFS/VM file size limit and rlimit */
@@ -1468,20 +1471,20 @@ int ll_setattr_raw(struct dentry *dentry, struct iattr *attr)ccc_inode_lsm_put (inode, lsm); } -/* clear size attr for released file +/* if not in HSM import mode, clear size attr for released file * we clear the attribute send to MDT in op_data, not the original * received from caller in attr which is used later to * decide return code */ -if (file_is_released && (attr->ia_valid &ATTR_SIZE )) +if (file_is_released && (attr->ia_valid &ATTR_SIZE ) && !hsm_import) op_data->op_attr.ia_valid &= ~ATTR_SIZE ; rc =ll_md_setattr (dentry, op_data, &mod);if (rc)GOTO (out, rc); -/* truncate failed, others succeed */ +/* truncate failed (only when non HSM import), others succeed */ if (file_is_released) { -if (attr->ia_valid &ATTR_SIZE ) +if ((attr->ia_valid &ATTR_SIZE ) && !hsm_import)GOTO (out, rc = -ENODATA );else GOTO (out, rc =0 );
@@ -1521,7 +1524,7 @@ out:if (!S_ISDIR (inode->i_mode)) {up_write (&lli->lli_trunc_sem);mutex_lock (&inode->i_mutex); -if (attr->ia_valid &ATTR_SIZE ) +if ((attr->ia_valid &ATTR_SIZE ) && !hsm_import)inode_dio_wait (inode); }
@@ -1556,7 +1559,7 @@ int ll_setattr(struct dentry *de, struct iattr *attr) !(attr->ia_valid & ATTR_KILL_SGID)) attr->ia_valid |=ATTR_KILL_SGID ; -return ll_setattr_raw (de, attr); +return ll_setattr_raw (de, attr, false); } int ll_statfs_internal(struct super_block *sb, struct obd_statfs *osfs,
drivers/staging/lustre/lustre/ptlrpc/wiretest.c +40 -0
@@ -4422,6 +4422,46 @@ void lustre_assert_wire_constants(void)LASSERTF ((int )sizeof (((struct hsm_user_request *)0 )->hur_user_item) ==0 ,"found %lld\n" , (long long )(int )sizeof (((struct hsm_user_request *)0 )->hur_user_item)); +/* Checks for struct hsm_user_import */ +LASSERTF ((int )sizeof (struct hsm_user_import ) ==48 ,"found %lld\n" , + (long long )(int )sizeof (struct hsm_user_import )); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_size) ==0 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_size)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_size) ==8 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_size)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_uid) ==32 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_uid)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_uid) ==4 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_uid)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_gid) ==36 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_gid)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_gid) ==4 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_gid)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_mode) ==40 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_mode)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_mode) ==4 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_mode)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_atime) ==8 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_atime)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_atime) ==8 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_atime)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_atime_ns) ==24 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_atime_ns)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_atime_ns) ==4 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_atime_ns)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_mtime) ==16 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_mtime)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_mtime) ==8 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_mtime)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_mtime_ns) ==28 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_mtime_ns)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_mtime_ns) ==4 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_mtime_ns)); +LASSERTF ((int )offsetof(struct hsm_user_import , hui_archive_id) ==44 ,"found %lld\n" , + (long long )(int )offsetof(struct hsm_user_import , hui_archive_id)); +LASSERTF ((int )sizeof (((struct hsm_user_import *)0 )->hui_archive_id) ==4 ,"found %lld\n" , + (long long )(int )sizeof (((struct hsm_user_import *)0 )->hui_archive_id)); +/* Checks for struct update_buf */ LASSERTF ((int )sizeof (struct update_buf ) ==8 ,"found %lld\n" , (long long )(int )sizeof (struct update_buf ));
Signature · 1 line
1.7.9.5
PATCH 31/40 staging/lustre/lnet: Add LNet Router Priority parameter
In reply to Peng TaoFrom: Doug Oucharek <[email protected]>
This change adds a priority parameter to the route module settings. This paramter can be >= 0. Like hops, the lower the prioirty number the higher the priority. So lower numbered priorities will be selected over higher numbers.
Lustre-change: http://review.whamcloud.com/5663
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2934
--- .../lustre/include/linux/libcfs/libcfs_ioctl.h | 1 + .../staging/lustre/include/linux/lnet/lib-lnet.h | 5 ++- .../staging/lustre/include/linux/lnet/lib-types.h | 2 +- drivers/staging/lustre/lnet/lnet/api-ni.c | 5 ++- drivers/staging/lustre/lnet/lnet/config.c | 39 +++++++++++++++++++- drivers/staging/lustre/lnet/lnet/lib-move.c | 6 +++ drivers/staging/lustre/lnet/lnet/router.c | 19 ++++++---- drivers/staging/lustre/lnet/lnet/router_proc.c | 16 ++++---- 8 files changed, 72 insertions(+), 21 deletions(-)
drivers/staging/lustre/include/linux/libcfs/libcfs_ioctl.h +1 -0
@@ -69,6 +69,7 @@ struct libcfs_ioctl_data {char ioc_bulk[0 ]; }; +#define ioc_priority ioc_u32[0]struct libcfs_ioctl_hdr {__u32 ioc_len;
drivers/staging/lustre/include/linux/lnet/lib-lnet.h +3 -2
@@ -650,12 +650,13 @@ extern lnet_ni_t *lnet_net2ni(__u32 net);int lnet_notify (lnet_ni_t *ni,lnet_nid_t peer,int alive,cfs_time_t when);void lnet_notify_locked (lnet_peer_t *lp,int notifylnd,int alive,cfs_time_t when); -int lnet_add_route (__u32 net,unsigned int hops,lnet_nid_t gateway_nid); +int lnet_add_route (__u32 net,unsigned int hops,lnet_nid_t gateway_nid, +unsigned int priority);int lnet_check_routes (void );int lnet_del_route (__u32 net,lnet_nid_t gw_nid);void lnet_destroy_routes (void );int lnet_get_route (int idx,__u32 *net,__u32 *hops, -lnet_nid_t *gateway,__u32 *alive); +lnet_nid_t *gateway,__u32 *alive,__u32 *priority);void lnet_proc_init (void );void lnet_proc_fini (void );int lnet_rtrpools_alloc (int im_a_router);
drivers/staging/lustre/include/linux/lnet/lib-types.h +1 -1
@@ -478,7 +478,6 @@ typedef struct lnet_peer {lnet_rc_data_t *lp_rcd;/* router checker state */ } lnet_peer_t; -/* peer hash size */ #define LNET_PEER_HASH_BITS 9#define LNET_PEER_HASH_SIZE (1 << LNET_PEER_HASH_BITS)
@@ -504,6 +503,7 @@ typedef struct {int lr_seq;/* sequence for round-robin */ unsigned int lr_downis;/* number of down NIs */ unsigned int lr_hops;/* how far I am */ +unsigned int lr_priority;/* route priority */ } lnet_route_t;#define LNET_REMOTE_NETS_HASH_DEFAULT (1U << 7)
drivers/staging/lustre/lnet/lnet/api-ni.c +3 -2
@@ -1436,7 +1436,7 @@ LNetCtl(unsigned int cmd, void *arg)case IOC_LIBCFS_ADD_ROUTE : rc =lnet_add_route (data->ioc_net, data->ioc_count, - data->ioc_nid); + data->ioc_nid, data->ioc_priority);return (rc !=0 ) ? rc :lnet_check_routes ();case IOC_LIBCFS_DEL_ROUTE :
@@ -1445,7 +1445,8 @@ LNetCtl(unsigned int cmd, void *arg)case IOC_LIBCFS_GET_ROUTE :return lnet_get_route (data->ioc_count, &data->ioc_net, &data->ioc_count, - &data->ioc_nid, &data->ioc_flags); + &data->ioc_nid, &data->ioc_flags, + &data->ioc_priority);case IOC_LIBCFS_NOTIFY_ROUTER : return lnet_notify(NULL, data->ioc_nid, data->ioc_flags, cfs_time_current() -
drivers/staging/lustre/lnet/lnet/config.c +38 -1
@@ -603,6 +603,37 @@ lnet_parse_hops(char *str, unsigned int *hops) *hops > 0 && *hops < 256); } +#define LNET_PRIORITY_SEPARATOR (':') + +int +lnet_parse_priority (char *str,unsigned int *priority,char **token) +{ +int nob; +char *sep; +int len; + + sep =strchr (str,LNET_PRIORITY_SEPARATOR ); +if (sep ==NULL ) { + *priority =0 ; +return 0 ; + } + len =strlen (sep +1 ); + +if ((sscanf ((sep+1 ),"%u%n" , priority, &nob) <1 ) || (len != nob)) { +/* Update the caller's token pointer so it treats the found +priority as the token to report in the error message. */ + *token += sep - str +1 ; +return -1 ; + } + +CDEBUG (D_NET ,"gateway %s, priority %d, nob %d\n" , str, *priority, nob); + +/* +* Change priority separator to \0 to be able to parse NID +*/ + *sep ='\0' ; +return 0 ; +}int lnet_parse_route (char *str,int *im_a_router)
@@ -624,6 +655,7 @@ lnet_parse_route(char *str, int *im_a_router)int myrc =-1 ;unsigned int hops;int got_hops =0 ; +unsigned int priority =0 ;INIT_LIST_HEAD (&gateways);INIT_LIST_HEAD (&nets);
@@ -691,6 +723,11 @@ lnet_parse_route(char *str, int *im_a_router) LNET_NETTYP(net) == LOLND) goto token_error; }else { + rc =lnet_parse_priority (ltb->ltb_text, + &priority, &token); +if (rc <0 ) + goto token_error; + nid =libcfs_str2nid (ltb->ltb_text);if (nid ==LNET_NID_ANY ||LNET_NETTYP (LNET_NIDNET (nid)) ==LOLND )
@@ -720,7 +757,7 @@ lnet_parse_route(char *str, int *im_a_router)continue ; } - rc =lnet_add_route (net, hops, nid); + rc =lnet_add_route (net, hops, nid, priority);if (rc !=0 ) { CERROR("Can't create route to %s via %s\n", libcfs_net2str(net),
drivers/staging/lustre/lnet/lnet/lib-move.c +6 -0
@@ -1074,6 +1074,12 @@ lnet_compare_routes(lnet_route_t *r1, lnet_route_t *r2)lnet_peer_t *p1 = r1->lr_gateway;lnet_peer_t *p2 = r2->lr_gateway; +if (r1->lr_priority < r2->lr_priority) +return 1 ; + +if (r1->lr_priority > r2->lr_priority) +return -1 ; +if (r1->lr_hops < r2->lr_hops)return 1 ;
drivers/staging/lustre/lnet/lnet/router.c +11 -8
@@ -301,7 +301,8 @@ lnet_add_route_to_rnet (lnet_remotenet_t *rnet, lnet_route_t *route) }int -lnet_add_route (__u32 net,unsigned int hops,lnet_nid_t gateway) +lnet_add_route (__u32 net,unsigned int hops,lnet_nid_t gateway, +unsigned int priority) {struct list_head *e;lnet_remotenet_t *rnet;
@@ -311,8 +312,8 @@ lnet_add_route (__u32 net, unsigned int hops, lnet_nid_t gateway)int add_route;int rc; -CDEBUG (D_NET ,"Add route: net %s hops %u gw %s\n" , -libcfs_net2str (net), hops,libcfs_nid2str (gateway)); +CDEBUG (D_NET ,"Add route: net %s hops %u priority %u gw %s\n" , +libcfs_net2str (net), hops, priority,libcfs_nid2str (gateway)); if (gateway == LNET_NID_ANY || LNET_NETTYP(LNET_NIDNET(gateway)) == LOLND ||
@@ -342,6 +343,7 @@ lnet_add_route (__u32 net, unsigned int hops, lnet_nid_t gateway) rnet->lrn_net = net; route->lr_hops = hops; route->lr_net = net; + route->lr_priority = priority;lnet_net_lock (LNET_LOCK_EX );
@@ -552,7 +554,7 @@ lnet_destroy_routes (void)int lnet_get_route (int idx,__u32 *net,__u32 *hops, -lnet_nid_t *gateway,__u32 *alive) +lnet_nid_t *gateway,__u32 *alive,__u32 *priority) {struct list_head *e1;struct list_head *e2;
@@ -574,10 +576,11 @@ lnet_get_route(int idx, __u32 *net, __u32 *hops, lr_list);if (idx-- ==0 ) { - *net = rnet->lrn_net; - *hops = route->lr_hops; - *gateway = route->lr_gateway->lp_nid; - *alive = route->lr_gateway->lp_alive; + *net = rnet->lrn_net; + *hops = route->lr_hops; + *priority = route->lr_priority; + *gateway = route->lr_gateway->lp_nid; + *alive = route->lr_gateway->lp_alive;lnet_net_unlock (cpt);return 0 ; }
drivers/staging/lustre/lnet/lnet/router_proc.c +9 -7
@@ -174,8 +174,8 @@ int LL_PROC_PROTO(proc_lnet_routes) the_lnet.ln_routing ? "enabled" : "disabled");LASSERT (tmpstr + tmpsiz - s >0 ); - s +=snprintf (s, tmpstr + tmpsiz - s,"%-8s %4s %7s %s\n" , -"net" ,"hops" ,"state" ,"router" ); + s +=snprintf (s, tmpstr + tmpsiz - s,"%-8s %4s %8s %7s %s\n" , +"net" ,"hops" ,"priority" ,"state" ,"router" );LASSERT (tmpstr + tmpsiz - s >0 );lnet_net_lock (0 );
@@ -229,14 +229,16 @@ int LL_PROC_PROTO(proc_lnet_routes) }if (route !=NULL ) { -__u32 net = rnet->lrn_net; -unsigned int hops = route->lr_hops; -lnet_nid_t nid = route->lr_gateway->lp_nid; -int alive = route->lr_gateway->lp_alive; +__u32 net = rnet->lrn_net; +unsigned int hops = route->lr_hops; +unsigned int priority = route->lr_priority; +lnet_nid_t nid = route->lr_gateway->lp_nid; +int alive = route->lr_gateway->lp_alive; s +=snprintf (s, tmpstr + tmpsiz - s, -"%-8s %4u %7s %s\n" , +"%-8s %4u %8u %7s %s\n" ,libcfs_net2str (net), hops, + priority, alive ?"up" :"down" ,libcfs_nid2str (nid));LASSERT (tmpstr + tmpsiz - s >0 );
Signature · 1 line
1.7.9.5
PATCH 33/40 staging/lustre/target: move OUT to the unified target code
In reply to Peng TaoFrom: Mikhail Pershin <[email protected]>
- Move OUT handler to the unified target code, so it can be used by both MDS and OST.
- Make OFD able to handle OUT requests.
- Rename MDS_MDS_PORTAL to the OUT_PORTAL and MDS_OUT_... defines just OUT_... since they are independent from MDS now.
Lustre-change: http://review.whamcloud.com/6763
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3467
--- .../lustre/lustre/include/lustre/lustre_idl.h | 3 +- drivers/staging/lustre/lustre/include/lustre_net.h | 8 ++--- drivers/staging/lustre/lustre/mdc/mdc_request.c | 32 +------------------- .../staging/lustre/lustre/obdecho/echo_client.c | 2 +- 4 files changed, 7 insertions(+), 38 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +1 -2
@@ -131,8 +131,7 @@//#define PTLBD_BULK_PORTAL 21 #define MDS_SETATTR_PORTAL 22#define MDS_READPAGE_PORTAL 23 -#define MDS_MDS_PORTAL 24 - +#define OUT_PORTAL 24#define MGC_REPLY_PORTAL 25#define MGS_REQUEST_PORTAL 26#define MGS_REPLY_PORTAL 27
drivers/staging/lustre/lustre/include/lustre_net.h +4 -4
@@ -370,8 +370,8 @@* include linkea (4K maxim), together with other updates, we set it to 9K: * lustre_msg + ptlrpc_body + UPDATE_BUF_SIZE (8K) */ -#define MDS_OUT_MAXREQSIZE (9 * 1024) -#define MDS_OUT_MAXREPSIZE MDS_MAXREPSIZE +#define OUT_MAXREQSIZE (9 * 1024) +#define OUT_MAXREPSIZE MDS_MAXREPSIZE/** MDS_BUFSIZE = max_reqsize (w/o LOV EA) + max sptlrpc payload size */ #define MDS_BUFSIZE max(MDS_MAXREQSIZE + SPTLRPC_MAX_PAYLOAD, \
@@ -397,11 +397,11 @@ 160 * 1024)/** -* MDS_OUT_BUFSIZE = max_out_reqsize + max sptlrpc payload (~1K) which is +* OUT_BUFSIZE = max_out_reqsize + max sptlrpc payload (~1K) which is * about 10K, for the same reason as MDS_REG_BUFSIZE, we also give some * extra bytes to each request buffer to improve buffer utilization rate. */ -#define MDS_OUT_BUFSIZE max(MDS_OUT_MAXREQSIZE + SPTLRPC_MAX_PAYLOAD, \ +#define OUT_BUFSIZE max(OUT_MAXREQSIZE + SPTLRPC_MAX_PAYLOAD, \ 24 * 1024)/** FLD_MAXREQSIZE == lustre_msg + __u32 padding + ptlrpc_body + opc */
drivers/staging/lustre/lustre/mdc/mdc_request.c +1 -31
@@ -2098,15 +2098,6 @@ int mdc_set_info_async(const struct lu_env *env,sptlrpc_import_flush_my_ctx (imp);return 0 ; } -if (KEY_IS (KEY_MDS_CONN )) { -/* mds-mds import */ -spin_lock (&imp->imp_lock); - imp->imp_server_timeout =1 ; -spin_unlock (&imp->imp_lock); - imp->imp_client->cli_request_portal =MDS_MDS_PORTAL ; -CDEBUG (D_OTHER ,"%s: timeout / 2\n" , exp->exp_obd->obd_name); -return 0 ; - }if (KEY_IS (KEY_CHANGELOG_CLEAR )) { rc =do_set_info_async (imp,MDS_SET_INFO ,LUSTRE_MDS_VERSION , keylen, key, vallen, val, set);
@@ -2616,27 +2607,6 @@ static int mdc_renew_capa(struct obd_export *exp, struct obd_capa *oc,return 0 ; } -static int mdc_connect (const struct lu_env *env, -struct obd_export **exp, -struct obd_device *obd,struct obd_uuid *cluuid, -struct obd_connect_data *data, -void *localdata) -{ -struct obd_import *imp = obd->u.cli.cl_import; - -/* mds-mds import features */ -if (data && (data->ocd_connect_flags &OBD_CONNECT_MDS_MDS )) { -spin_lock (&imp->imp_lock); - imp->imp_server_timeout =1 ; -spin_unlock (&imp->imp_lock); - imp->imp_client->cli_request_portal =MDS_MDS_PORTAL ; -CDEBUG (D_OTHER ,"%s: Set 'mds' portal and timeout\n" , - obd->obd_name); - } - -return client_connect_import (env, exp, obd, cluuid, data,NULL ); -} -struct obd_ops mdc_obd_ops = { .o_owner =THIS_MODULE , .o_setup = mdc_setup,
@@ -2644,7 +2614,7 @@ struct obd_ops mdc_obd_ops = { .o_cleanup = mdc_cleanup, .o_add_conn = client_import_add_conn, .o_del_conn = client_import_del_conn, - .o_connect = mdc_connect, + .o_connect = client_connect_import, .o_disconnect = client_disconnect_export, .o_iocontrol = mdc_iocontrol, .o_set_info_async = mdc_set_info_async,
drivers/staging/lustre/lustre/obdecho/echo_client.c +1 -1
@@ -2758,7 +2758,7 @@ echo_client_iocontrol(unsigned int cmd, struct obd_export *exp, int len,if (env ==NULL )return -ENOMEM ; - rc =lu_env_init (env,LCT_DT_THREAD ); + rc =lu_env_init (env,LCT_DT_THREAD |LCT_MD_THREAD );if (rc)GOTO (out, rc = -ENOMEM );
Signature · 1 line
1.7.9.5
PATCH 34/40 staging/lustre/seq: unified SEQ handler
In reply to Peng TaoFrom: Mikhail Pershin <[email protected]>
move SEQ handler from MDT to the FID module so it can be used from any target.
Lustre-change: http://review.whamcloud.com/6765
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3467
--- drivers/staging/lustre/lustre/include/lustre_fid.h | 6 ------ 1 file changed, 6 deletions(-)
drivers/staging/lustre/lustre/include/lustre_fid.h +0 -6
@@ -431,12 +431,6 @@ struct lu_server_seq {struct seq_server_site *lss_site; }; -struct com_thread_info ; -int seq_query (struct com_thread_info *info); - -struct ptlrpc_request ; -int seq_handle (struct ptlrpc_request *req); -/* Server methods */ int seq_server_init(struct lu_server_seq *seq,
Signature · 1 line
1.7.9.5
PATCH 36/40 staging/lustre/llite: pass correct pointer to obd_iocontrol()
In reply to Peng TaoFrom: "John L. Hammond" <[email protected]>
In copy_and_ioctl() use the kernel space copy as the karg to obd_iocontrol().
Lustre-change: http://review.whamcloud.com/6274
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3283
--- drivers/staging/lustre/lustre/llite/dir.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
drivers/staging/lustre/lustre/llite/dir.c +14 -9
@@ -1048,20 +1048,25 @@ progress: } -static int copy_and_ioctl (int cmd,struct obd_export *exp,void *data,int len) +static int copy_and_ioctl (int cmd,struct obd_export *exp, +const void __user *data,size_t size) { -void *ptr; +void *copy;int rc; -OBD_ALLOC (ptr, len); -if (ptr ==NULL ) +OBD_ALLOC (copy, size); +if (copy ==NULL )return -ENOMEM ; -if (copy_from_user (ptr, data, len)) { -OBD_FREE (ptr, len); -return -EFAULT ; + +if (copy_from_user (copy, data, size)) { + rc = -EFAULT ; + goto out; } - rc =obd_iocontrol (cmd, exp, len, data,NULL ); -OBD_FREE (ptr, len); + + rc =obd_iocontrol (cmd, exp, size, copy,NULL ); +out: +OBD_FREE (copy, size); +return rc; }
Signature · 1 line
1.7.9.5
PATCH 35/40 staging/lustre/llite: remove ll_d_root_ops
In reply to Peng TaoFrom: Lai Siyao <[email protected]>
Mnt root dentry will never be revalidated, but its d_op->d_compare will be called for its children, to simplify code, we use the same ll_d_ops as normal dentries.
But its attribute may be invalid before access, this won't cause any issue because it always exists, and the only operation depends on its attribute is .permission, which has revalidated it in lustre code.
So it's okay to remove ll_d_root_ops, and remove unnecessary checks in lookup/revalidate/statahead.
This patch also includes several small fixes:
* don't d_add() for create only files, because the dentry is fake, and will be released right after use.
* alloc ll_dentry_data may fail, add check for this.
Lustre-change: http://review.whamcloud.com/6797
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3486
--- drivers/staging/lustre/lustre/llite/dcache.c | 41 +++++--------------- .../staging/lustre/lustre/llite/llite_internal.h | 5 ++- drivers/staging/lustre/lustre/llite/llite_lib.c | 11 +----- drivers/staging/lustre/lustre/llite/llite_nfs.c | 6 +-- drivers/staging/lustre/lustre/llite/namei.c | 33 +++++++--------- drivers/staging/lustre/lustre/llite/statahead.c | 7 ++-- 6 files changed, 35 insertions(+), 68 deletions(-)
drivers/staging/lustre/lustre/llite/dcache.c +9 -32
@@ -37,6 +37,7 @@#include <linux/fs.h> #include <linux/sched.h> #include <linux/quotaops.h> +#include <linux/kernel.h> #define DEBUG_SUBSYSTEM S_LLITE
@@ -176,7 +177,7 @@ static int ll_ddelete(const struct dentry *de)return 0 ; } -static int ll_set_dd (struct dentry *de) +int ll_d_init (struct dentry *de) {LASSERT (de !=NULL );
@@ -190,40 +191,22 @@ static int ll_set_dd(struct dentry *de)OBD_ALLOC_PTR (lld);if (likely (lld !=NULL )) {spin_lock (&de->d_lock); -if (likely (de->d_fsdata ==NULL )) +if (likely (de->d_fsdata ==NULL )) { de->d_fsdata = lld; -else +__d_lustre_invalidate (de); + }else {OBD_FREE_PTR (lld); + }spin_unlock (&de->d_lock); }else {return -ENOMEM ; } } +LASSERT (de->d_op == &ll_d_ops);return 0 ; } -int ll_dops_init (struct dentry *de,int block,int init_sa) -{ -struct ll_dentry_data *lld =ll_d2d (de); -int rc =0 ; - -if (lld ==NULL && block !=0 ) { - rc =ll_set_dd (de); -if (rc) -return rc; - - lld =ll_d2d (de); - } - -if (lld !=NULL && init_sa !=0 ) - lld->lld_sa_generation =0 ; - -/* kernel >= 2.6.38 d_op is set in d_alloc() */ -LASSERT (de->d_op == &ll_d_ops); -return rc; -} -void ll_intent_drop_lock (struct lookup_intent *it) {if (it->it_op && it->d.lustre.it_lock_mode) {
@@ -358,6 +341,8 @@ int ll_revalidate_it(struct dentry *de, int lookup_flags,CDEBUG (D_VFSTRACE ,"VFS Op:name=%s,intent=%s\n" , de->d_name.name,LL_IT2STR (it)); +LASSERT (de != de->d_sb->s_root); +if (de->d_inode ==NULL ) {__u64 ibits;
@@ -382,14 +367,6 @@ int ll_revalidate_it(struct dentry *de, int lookup_flags,if (d_mountpoint (de))GOTO (out_sa, rc =1 ); -/* need to get attributes in case root got changed from other client */ -if (de == de->d_sb->s_root) { - rc =__ll_inode_revalidate_it (de, it,MDS_INODELOCK_LOOKUP ); -if (rc ==0 ) - rc =1 ; -GOTO (out_sa, rc); - } - exp =ll_i2mdexp (de->d_inode);OBD_FAIL_TIMEOUT (OBD_FAIL_MDC_REVALIDATE_PAUSE ,5 );
drivers/staging/lustre/lustre/llite/llite_internal.h +3 -2
@@ -816,7 +816,7 @@ int ll_lease_close(struct obd_client_handle *och, struct inode *inode,/* llite/dcache.c */ -int ll_dops_init (struct dentry *de,int block,int init_sa); +int ll_d_init (struct dentry *de);extern struct dentry_operations ll_d_ops;void ll_intent_drop_lock (struct lookup_intent *);void ll_intent_release (struct lookup_intent *);
@@ -1308,7 +1308,8 @@ ll_statahead_mark(struct inode *dir, struct dentry *dentry)if (lli->lli_opendir_pid !=current_pid ())return ; -if (sai !=NULL && ldd !=NULL ) +LASSERT (ldd !=NULL ); +if (sai !=NULL ) ldd->lld_sa_generation = sai->sai_generation; }
drivers/staging/lustre/lustre/llite/llite_lib.c +2 -9
@@ -154,11 +154,6 @@ void ll_free_sbi(struct super_block *sb) } } -static struct dentry_operations ll_d_root_ops = { - .d_compare = ll_dcompare, - .d_revalidate = ll_revalidate_nd, -}; -static int client_common_fill_super (struct super_block *sb,char *md,char *dt,struct vfsmount *mnt) {
@@ -577,10 +572,6 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt,GOTO (out_root, err = -ENOMEM ); } -/* kernel >= 2.6.38 store dentry operations in sb->s_d_op. */ -d_set_d_op (sb->s_root, &ll_d_root_ops); - sb->s_d_op = &ll_d_ops; - sbi->ll_sdev_orig = sb->s_dev;/* We set sb->s_dev equal on all lustre clients in order to support
@@ -1011,6 +1002,8 @@ int ll_fill_super(struct super_block *sb, struct vfsmount *mnt)GOTO (out_free, err); sb->s_bdi = &lsi->lsi_bdi; +/* kernel >= 2.6.38 store dentry operations in sb->s_d_op. */ + sb->s_d_op = &ll_d_ops;/* Generate a string unique to this super, in case some joker tries to mount the same fs at two mount points.
drivers/staging/lustre/lustre/llite/llite_nfs.c +3 -3
@@ -167,10 +167,10 @@ ll_iget_for_nfs(struct super_block *sb, struct lu_fid *fid, struct lu_fid *paren } result =d_obtain_alias (inode); -if (IS_ERR (result)) +if (IS_ERR (result)) { +iput (inode);return result; - -ll_dops_init (result,1 ,0 ); + }return result; }
drivers/staging/lustre/lustre/llite/namei.c +14 -19
@@ -398,11 +398,16 @@ static struct dentry *ll_find_alias(struct inode *inode, struct dentry *dentry)struct dentry *ll_splice_alias (struct inode *inode,struct dentry *de) {struct dentry *new; +int rc;if (inode) { new =ll_find_alias (inode, de);if (new) { -ll_dops_init (new,1 ,1 ); + rc =ll_d_init (new); +if (rc <0 ) { +dput (new); +return ERR_PTR (rc); + }d_move (new, de);iput (inode); CDEBUG(D_DENTRY,
@@ -411,8 +416,9 @@ struct dentry *ll_splice_alias(struct inode *inode, struct dentry *de)return new; } } -ll_dops_init (de,1 ,1 ); -__d_lustre_invalidate (de); + rc =ll_d_init (de); +if (rc <0 ) +return ERR_PTR (rc);d_add (de, inode);CDEBUG (D_DENTRY ,"Add dentry %p inode %p refc %d flags %#x\n" , de, de->d_inode,d_count (de), de->d_flags);
@@ -453,8 +459,11 @@ int ll_lookup_it_finish(struct ptlrpc_request *request,/* Only hash *de if it is unhashed (new dentry). * Atoimc_open may passin hashed dentries for open. */ -if (d_unhashed (*de)) +if (d_unhashed (*de)) { *de =ll_splice_alias (inode, *de); +if (IS_ERR (*de)) +return PTR_ERR (*de); + }if (!it_disposition (it,DISP_LOOKUP_NEG )) {/* we have lookup look - unhide dentry */
@@ -503,16 +512,6 @@ static struct dentry *ll_lookup_it(struct inode *parent, struct dentry *dentry,ll_frob_intent (&it, &lookup_it); -/* As do_lookup is called before follow_mount, root dentry may be left -* not valid, revalidate it here. */ -if (parent->i_sb->s_root && (parent->i_sb->s_root->d_inode == parent) && - (it->it_op & (IT_OPEN |IT_CREAT ))) { - rc =ll_inode_revalidate_it (parent->i_sb->s_root, it, -MDS_INODELOCK_LOOKUP ); -if (rc) -return ERR_PTR (rc); - } -if (it->it_op ==IT_GETATTR ) { rc =ll_statahead_enter (parent, &dentry,0 );if (rc ==1 ) {
@@ -582,12 +581,8 @@ static struct dentry *ll_lookup_nd(struct inode *parent, struct dentry *dentry, parent->i_generation, parent, flags);/* Optimize away (CREATE && !OPEN). Let .create handle the race. */ -if ((flags &LOOKUP_CREATE ) && !(flags &LOOKUP_OPEN )) { -ll_dops_init (dentry,1 ,1 ); -__d_lustre_invalidate (dentry); -d_add (dentry,NULL ); +if ((flags &LOOKUP_CREATE ) && !(flags &LOOKUP_OPEN ))return NULL ; - }if (flags & (LOOKUP_PARENT |LOOKUP_OPEN |LOOKUP_CREATE )) itp =NULL ;
drivers/staging/lustre/lustre/llite/statahead.c +4 -3
@@ -877,9 +877,6 @@ static int do_sa_revalidate(struct inode *dir, struct ll_sa_entry *entry,if (d_mountpoint (dentry))return 1 ; -if (unlikely (dentry == dentry->d_sb->s_root)) -return 1 ; - entry->se_inode =igrab (inode); rc =md_revalidate_lock (ll_i2mdexp (dir), &it,ll_inode2fid (inode),NULL );if (rc ==1 ) {
@@ -1590,6 +1587,10 @@ int do_statahead_enter(struct inode *dir, struct dentry **dentryp,if ((*dentryp)->d_inode ==NULL ) { *dentryp =ll_splice_alias (inode, *dentryp); +if (IS_ERR (*dentryp)) { +ll_sai_unplug (sai, entry); +return PTR_ERR (*dentryp); + } }else if ((*dentryp)->d_inode != inode) {/* revalidate, but inode is recreated */ CDEBUG(D_READA,
Signature · 1 line
1.7.9.5
PATCH 38/40 staging/lustre/mgs: set_param -P option that sets value permanently
In reply to Peng TaoFrom: Artem Blagodarenko <[email protected]>
set_param and conf_param have different syntaxes. Also conf_param has unimplemented paths and no wildcarding support.
This patch adds set_param -P option, that replaces the whole conf_param "direct" proc access with a simple upcall-type mechanism from the MGC. Option conf_param is saved now for compatibility.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3155
Lustre-change: http://review.whamcloud.com/6025
--- drivers/staging/lustre/lustre/include/lustre_cfg.h | 2 + .../staging/lustre/lustre/include/lustre_disk.h | 2 + drivers/staging/lustre/lustre/include/obd_class.h | 9 +- drivers/staging/lustre/lustre/llite/llite_lib.c | 5 +- drivers/staging/lustre/lustre/mgc/mgc_request.c | 89 +++++++++++++++++--- .../staging/lustre/lustre/obdclass/obd_config.c | 50 +++++++++-- 6 files changed, 134 insertions(+), 23 deletions(-)
drivers/staging/lustre/lustre/include/lustre_cfg.h +2 -0
@@ -88,6 +88,8 @@ enum lcfg_command_type {LCFG_SET_LDLM_TIMEOUT =0x00ce030 ,/**< set ldlm_timeout */ LCFG_PRE_CLEANUP =0x00cf031 ,/**< call type-specific pre * cleanup cleanup */ + LCFG_SET_PARAM = 0x00ce032, /**< use set_param syntax to set + *a proc parameters */ }; struct lustre_cfg_bufs {
drivers/staging/lustre/lustre/include/lustre_disk.h +2 -0
@@ -98,6 +98,8 @@#define LDD_F_IR_CAPABLE 0x2000/** the MGS refused to register the target. */ #define LDD_F_ERROR 0x4000 +/** process at lctl conf_param */ +#define LDD_F_PARAM2 0x8000/* opc for target register */ #define LDD_F_OPC_REG 0x10000000
drivers/staging/lustre/lustre/include/obd_class.h +7 -2
@@ -175,9 +175,13 @@ enum {CONFIG_T_CONFIG =0 ,CONFIG_T_SPTLRPC =1 ,CONFIG_T_RECOVER =2 , -CONFIG_T_MAX =3 +CONFIG_T_PARAMS =3 , +CONFIG_T_MAX =4 }; +#define PARAMS_FILENAME "params" +#define LCTL_UPCALL "lctl" +/* list of active configuration logs */ struct config_llog_data { struct ldlm_res_id cld_resid;
@@ -185,7 +189,8 @@ struct config_llog_data {struct list_head cld_list_chain;atomic_t cld_refcount;struct config_llog_data *cld_sptlrpc;/* depended sptlrpc log */ -struct config_llog_data *cld_recover;/* imperative recover log */ +struct config_llog_data *cld_params;/* common parameters log */ +struct config_llog_data *cld_recover;/* imperative recover log */ struct obd_export *cld_mgcexp;struct mutex cld_lock;int cld_type;
drivers/staging/lustre/lustre/llite/llite_lib.c +4 -1
@@ -1058,7 +1058,7 @@ out_free:void ll_put_super (struct super_block *sb) { -struct config_llog_instance cfg; +struct config_llog_instance cfg, params_cfg;struct obd_device *obd;struct lustre_sb_info *lsi =s2lsi (sb);struct ll_sb_info *sbi =ll_s2sbi (sb);
@@ -1072,6 +1072,9 @@ void ll_put_super(struct super_block *sb) cfg.cfg_instance = sb;lustre_end_log (sb, profilenm, &cfg); + params_cfg.cfg_instance = sb; +lustre_end_log (sb,PARAMS_FILENAME , ¶ms_cfg); +if (sbi->ll_md_exp) { obd =class_exp2obd (sbi->ll_md_exp);if (obd)
drivers/staging/lustre/lustre/mgc/mgc_request.c +75 -14
180 lines not on this page · raw message
drivers/staging/lustre/lustre/obdclass/obd_config.c +44 -6
85 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH 40/40 staging/lustre/ptlrpc: flock deadlock detection does not work
In reply to Peng TaoFrom: Andriy Skulysh <[email protected]>
Flock deadlocks are checked on the first attempt to grant the flock only. If we try again to grant it after its blocking lock is cancelled, we don't check for deadlocks which also may exist.
Perform deadlock detection during reprocess
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-1715
Lustre-change: http://review.whamcloud.com/3553
--- .../lustre/lustre/include/lustre/lustre_idl.h | 5 ++- .../lustre/lustre/include/lustre_dlm_flags.h | 14 +++++-- drivers/staging/lustre/lustre/ldlm/ldlm_flock.c | 41 ++++++++++++++++++-- drivers/staging/lustre/lustre/llite/llite_lib.c | 3 +- .../lustre/lustre/obdclass/lprocfs_status.c | 1 + drivers/staging/lustre/lustre/ptlrpc/wiretest.c | 2 + 6 files changed, 57 insertions(+), 9 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +4 -1
@@ -1292,6 +1292,8 @@ extern void lustre_swab_ptlrpc_body(struct ptlrpc_body *pb);#define OBD_CONNECT_LIGHTWEIGHT 0x1000000000000ULL/* lightweight connection */ #define OBD_CONNECT_SHORTIO 0x2000000000000ULL/* short io */ #define OBD_CONNECT_PINGLESS 0x4000000000000ULL/* pings not required */ +#define OBD_CONNECT_FLOCK_DEAD 0x8000000000000ULL/* improved flock deadlock detection */ +/* XXX README XXX: * Please DO NOT add flag values here before first ensuring that this same * flag value is not in use on some other branch. Please clear any such
@@ -1329,7 +1331,8 @@ extern void lustre_swab_ptlrpc_body(struct ptlrpc_body *pb); OBD_CONNECT_EINPROGRESS | \ OBD_CONNECT_LIGHTWEIGHT | OBD_CONNECT_UMASK | \ OBD_CONNECT_LVB_TYPE | OBD_CONNECT_LAYOUTLOCK |\ - OBD_CONNECT_PINGLESS | OBD_CONNECT_MAX_EASIZE) + OBD_CONNECT_PINGLESS | OBD_CONNECT_MAX_EASIZE |\ + OBD_CONNECT_FLOCK_DEAD)#define OST_CONNECT_SUPPORTED (OBD_CONNECT_SRVLOCK | OBD_CONNECT_GRANT | \ OBD_CONNECT_REQPORTAL | OBD_CONNECT_VERSION | \ OBD_CONNECT_TRUNCLOCK | OBD_CONNECT_INDEX | \
drivers/staging/lustre/lustre/include/lustre_dlm_flags.h +11 -3
@@ -35,10 +35,10 @@#ifndef LDLM_ALL_FLAGS_MASK /** l_flags bits marked as "all_flags" bits */ -#define LDLM_FL_ALL_FLAGS_MASK 0x00FFFFFFC08F132FULL +#define LDLM_FL_ALL_FLAGS_MASK 0x00FFFFFFC08F932FULL/** l_flags bits marked as "ast" bits */ -#define LDLM_FL_AST_MASK 0x0000000080000000ULL +#define LDLM_FL_AST_MASK 0x0000000080008000ULL/** l_flags bits marked as "blocked" bits */ #define LDLM_FL_BLOCKED_MASK 0x000000000000000EULL
@@ -56,7 +56,7 @@#define LDLM_FL_LOCAL_ONLY_MASK 0x00FFFFFF00000000ULL/** l_flags bits marked as "on_wire" bits */ -#define LDLM_FL_ON_WIRE_MASK 0x00000000C08F132FULL +#define LDLM_FL_ON_WIRE_MASK 0x00000000C08F932FULL/** extent, mode, or resource changed */ #define LDLM_FL_LOCK_CHANGED 0x0000000000000001ULL // bit 0
@@ -114,6 +114,12 @@#define ldlm_set_has_intent (_l) LDLM_SET_FLAG(( _l), 1ULL << 12)#define ldlm_clear_has_intent (_l) LDLM_CLEAR_FLAG((_l), 1ULL << 12) +/** flock deadlock detected */ +#define LDLM_FL_FLOCK_DEADLOCK 0x0000000000008000ULL // bit 15 +#define ldlm_is_flock_deadlock (_l) LDLM_TEST_FLAG(( _l), 1ULL << 15) +#define ldlm_set_flock_deadlock (_l) LDLM_SET_FLAG(( _l), 1ULL << 15) +#define ldlm_clear_flock_deadlock (_l) LDLM_CLEAR_FLAG((_l), 1ULL << 15) +/** discard (no writeback) on cancel */ #define LDLM_FL_DISCARD_DATA 0x0000000000010000ULL // bit 16#define ldlm_is_discard_data (_l) LDLM_TEST_FLAG(( _l), 1ULL << 16)
@@ -390,6 +396,7 @@ static int hf_lustre_ldlm_fl_ast_sent = -1;static int hf_lustre_ldlm_fl_replay =-1 ;static int hf_lustre_ldlm_fl_intent_only =-1 ;static int hf_lustre_ldlm_fl_has_intent =-1 ; +static int hf_lustre_ldlm_fl_flock_deadlock =-1 ;static int hf_lustre_ldlm_fl_discard_data =-1 ;static int hf_lustre_ldlm_fl_no_timeout =-1 ;static int hf_lustre_ldlm_fl_block_nowait =-1 ;
@@ -431,6 +438,7 @@ const value_string lustre_ldlm_flags_vals[] = { {LDLM_FL_REPLAY, "LDLM_FL_REPLAY"}, {LDLM_FL_INTENT_ONLY, "LDLM_FL_INTENT_ONLY"}, {LDLM_FL_HAS_INTENT, "LDLM_FL_HAS_INTENT"}, + {LDLM_FL_FLOCK_DEADLOCK, "LDLM_FL_FLOCK_DEADLOCK"}, {LDLM_FL_DISCARD_DATA, "LDLM_FL_DISCARD_DATA"}, {LDLM_FL_NO_TIMEOUT, "LDLM_FL_NO_TIMEOUT"}, {LDLM_FL_BLOCK_NOWAIT, "LDLM_FL_BLOCK_NOWAIT"},
drivers/staging/lustre/lustre/ldlm/ldlm_flock.c +37 -4
77 lines not on this page · raw message
drivers/staging/lustre/lustre/llite/llite_lib.c +2 -1
@@ -205,7 +205,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt,OBD_CONNECT_EINPROGRESS |OBD_CONNECT_JOBSTATS |OBD_CONNECT_LVB_TYPE |OBD_CONNECT_LAYOUTLOCK |OBD_CONNECT_PINGLESS | -OBD_CONNECT_MAX_EASIZE ; +OBD_CONNECT_MAX_EASIZE | +OBD_CONNECT_FLOCK_DEAD ;if (sbi->ll_flags &LL_SBI_SOM_PREVIEW ) data->ocd_connect_flags |=OBD_CONNECT_SOM ;
drivers/staging/lustre/lustre/obdclass/lprocfs_status.c +1 -0
@@ -531,6 +531,7 @@ static const char *obd_connect_names[] = {"lightweight_conn" ,"short_io" ,"pingless" , +"flock_deadlock" ,"unknown" , NULL };
drivers/staging/lustre/lustre/ptlrpc/wiretest.c +2 -0
@@ -1153,6 +1153,8 @@ void lustre_assert_wire_constants(void) OBD_CONNECT_SHORTIO);LASSERTF (OBD_CONNECT_PINGLESS ==0x4000000000000ULL ,"found 0x%.16llxULL\n" ,OBD_CONNECT_PINGLESS ); +LASSERTF (OBD_CONNECT_FLOCK_DEAD ==0x8000000000000ULL ,"found 0x%.16llxULL\n" , +OBD_CONNECT_FLOCK_DEAD );LASSERTF (OBD_CKSUM_CRC32 ==0x00000001UL ,"found 0x%.8xUL\n" , (unsigned )OBD_CKSUM_CRC32 ); LASSERTF(OBD_CKSUM_ADLER == 0x00000002UL, "found 0x%.8xUL\n",
Signature · 1 line
1.7.9.5
PATCH 39/40 staging/lustre/utils: HSM Posix CopyTool
In reply to Peng TaoFrom: JC Lafoucriere <[email protected]>
POSIX HSM CopyTool utils named lhsmtool_posix.
This user space command is the 'glue" between Lustre-HSM and a POSIX filesytem used as a backend.
The main functionalities implemented are:
daemon mode:
- archive: read in lustre write with POSIX backend
- restore: read in POSIX backend write to Lustre
- remove: remove an entry from backend
cmd line mode:
- import: create in lustre a released file from a backend file
- rebind: change the FID associated to a file in the backend
- maxseq: get the larger sequence of FID found in the backend
The 2 last options are used for disaster recovery mode
This tools is also used for all the non regression tests made in sanity-hsm.sh
Lustre-change: http://review.whamcloud.com/4737
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2062
--- .../lustre/lustre/include/lustre/lustre_user.h | 9 ++--- .../lustre/lustre/include/lustre/lustreapi.h | 36 +++++++++++++------- 2 files changed, 26 insertions(+), 19 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_user.h +3 -6
@@ -311,6 +311,9 @@ struct ost_id {#define LOV_ALL_STRIPES 0xffff/* only valid for directories */ #define LOV_V1_INSANE_STRIPE_COUNT 65532/* maximum stripe count bz13933 */ +#define XATTR_LUSTRE_PREFIX "lustre." +#define XATTR_LUSTRE_LOV XATTR_LUSTRE_PREFIX"lov" +#define lov_user_ost_data lov_user_ost_data_v1struct lov_user_ost_data_v1 {/* per-stripe data structure */ struct ost_id l_ost_oi;/* OST object ID */
@@ -1158,12 +1161,6 @@ struct hsm_progress {__u32 padding; }; -/** -* Use by copytool during any hsm request they handled. -* This structure is initialized by llapi_hsm_copy_start() -* which is an helper over the ioctl() interface -* Store Lustre, internal use only, data. -*/ struct hsm_copy {__u64 hc_data_version;__u16 hc_flags;
drivers/staging/lustre/lustre/include/lustre/lustreapi.h +23 -13
@@ -219,8 +219,8 @@ extern int llapi_lmv_get_uuids(int fd, struct obd_uuid *uuidp, int *mdt_count);extern int llapi_is_lustre_mnttype (const char *type);extern int llapi_search_ost (char *fsname,char *poolname,char *ostname);extern int llapi_get_obd_count (char *mnt,int *count,int is_mdt); -extern int parse_size (char *optarg,unsigned long long *size, -unsigned long long *size_units,int bytes_spec); +extern int llapi_parse_size (const char *optarg,unsigned long long *size, +unsigned long long *size_units,int bytes_spec);extern int llapi_search_mounts (const char *pathname,int index,char *mntdir,char *fsname);extern int llapi_search_fsname (const char *pathname,char *fsname);
@@ -298,18 +298,27 @@ extern int llapi_changelog_clear(const char *mdtname, const char *idstr,* priv is private state, managed internally by these functions */ struct hsm_copytool_private ; -extern int llapi_hsm_copytool_start (struct hsm_copytool_private **priv, -char *fsname,int flags, -int archive_count,int *archives); -extern int llapi_hsm_copytool_fini (struct hsm_copytool_private **priv); +struct hsm_copyaction_private ; + +extern int llapi_hsm_copytool_register (struct hsm_copytool_private **priv, +const char *mnt,int flags, +int archive_count,int *archives); +extern int llapi_hsm_copytool_unregister (struct hsm_copytool_private **priv);extern int llapi_hsm_copytool_recv (struct hsm_copytool_private *priv,struct hsm_action_list **hal,int *msgsize); -extern int llapi_hsm_copytool_free (struct hsm_action_list **hal); -extern int llapi_hsm_copy_start (char *mnt,struct hsm_copy *copy, -const struct hsm_action_item *hai); -extern int llapi_hsm_copy_end (char *mnt,struct hsm_copy *copy, -const struct hsm_progress *hp); -extern int llapi_hsm_progress (char *mnt,struct hsm_progress *hp); +extern void llapi_hsm_action_list_free (struct hsm_action_list **hal); +extern int llapi_hsm_action_begin (struct hsm_copyaction_private **hcp, +const struct hsm_copytool_private *ct_priv, +const struct hsm_action_item *hai, +bool is_error); +extern int llapi_hsm_action_end (struct hsm_copyaction_private **hcp, +const struct hsm_extent *he,int flags, +int errval); +extern int llapi_hsm_action_progress (struct hsm_copyaction_private *hcp, +const struct hsm_extent *he,int hp_flags); +extern int llapi_hsm_action_get_dfid (const struct hsm_copyaction_private *hcp, +lustre_fid *fid); +extern int llapi_hsm_action_get_fd (const struct hsm_copyaction_private *hcp); extern int llapi_hsm_import(const char *dst, int archive, const struct stat *st, unsigned long long stripe_size, int stripe_offset, int stripe_count, int stripe_pattern,
@@ -318,7 +327,8 @@ extern int llapi_hsm_import(const char *dst, int archive, const struct stat *st,/* HSM user interface */ extern struct hsm_user_request *llapi_hsm_user_request_alloc (int itemcount,int data_len); -extern int llapi_hsm_request (char *mnt,struct hsm_user_request *request); +extern int llapi_hsm_request (const char *path, +const struct hsm_user_request *request);extern int llapi_hsm_current_action (const char *path,struct hsm_current_action *hca);/** @} llapi */
Signature · 1 line
1.7.9.5
PATCH 37/40 staging/lustre/idl: remove LASSERT/CLASSERT from lustre_idl.h
In reply to Peng TaoFrom: Andreas Dilger <[email protected]>
Remove the usage of LASSERT() and CLASSERT() from lustre_idl.h, so that it is usable from userspace programs if needed. These have crept in over the years, but are not intended to be there.
The CLASSERT() checks for fid swabbing were largely redundant, and have been moved to lustre/fid/fid_handler.c.
The fid_is_sane() macro has been modified to remove the check that f_ver == 0, since this will eventually not be true, and the client or server should not crash if it sees such a FID during usage.
There are still a few LASSERTs that need to be removed when FID-on-OST is landed, but I don't want to remove them before that code lands.
There are also uses of CERROR() that could be removed at that time.
Lustre-change: http://review.whamcloud.com/5682
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-1606
--- .../lustre/lustre/include/lustre/lustre_idl.h | 43 ++++---------------- 1 file changed, 8 insertions(+), 35 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +8 -35
@@ -91,8 +91,8 @@#ifndef _LUSTRE_IDL_H_#define _LUSTRE_IDL_H_ -#if !defined(LASSERT ) && !defined(LPU64 ) -#include <linux/libcfs/libcfs.h> /* for LASSERT, LPUX64, etc */ +#if !defined(LPU64 ) +#include <linux/libcfs/libcfs.h> /* for LPUX64, etc */ #endif /* Defn's shared with user-space. */
@@ -231,7 +231,6 @@ static inline unsigned fld_range_is_any(const struct lu_seq_range *range)static inline void fld_range_set_type (struct lu_seq_range *range,unsigned flags) { -LASSERT (!(flags & ~LU_SEQ_RANGE_MASK )); range->lsr_flags |= flags; }
@@ -615,7 +614,6 @@ static inline obd_id fid_idif_id(obd_seq seq, __u32 oid, __u32 ver)/* extract ost index from IDIF FID */ static inline __u32 fid_idif_ost_idx (const struct lu_fid *fid) { -LASSERT (fid_is_idif (fid));return (fid_seq (fid) >>16 ) &0xffff ; }
@@ -833,11 +831,6 @@ static inline void lu_igif_build(struct lu_fid *fid, __u32 ino, __u32 gen)*/ static inline void fid_cpu_to_le (struct lu_fid *dst,const struct lu_fid *src) { -/* check that all fields are converted */ -CLASSERT (sizeof (*src) == -sizeof (fid_seq (src)) + -sizeof (fid_oid (src)) + -sizeof (fid_ver (src))); dst->f_seq =cpu_to_le64 (fid_seq (src)); dst->f_oid =cpu_to_le32 (fid_oid (src)); dst->f_ver =cpu_to_le32 (fid_ver (src));
@@ -845,11 +838,6 @@ static inline void fid_cpu_to_le(struct lu_fid *dst, const struct lu_fid *src)static inline void fid_le_to_cpu (struct lu_fid *dst,const struct lu_fid *src) { -/* check that all fields are converted */ -CLASSERT (sizeof (*src) == -sizeof (fid_seq (src)) + -sizeof (fid_oid (src)) + -sizeof (fid_ver (src))); dst->f_seq =le64_to_cpu (fid_seq (src)); dst->f_oid =le32_to_cpu (fid_oid (src)); dst->f_ver =le32_to_cpu (fid_ver (src));
@@ -857,11 +845,6 @@ static inline void fid_le_to_cpu(struct lu_fid *dst, const struct lu_fid *src)static inline void fid_cpu_to_be (struct lu_fid *dst,const struct lu_fid *src) { -/* check that all fields are converted */ -CLASSERT (sizeof (*src) == -sizeof (fid_seq (src)) + -sizeof (fid_oid (src)) + -sizeof (fid_ver (src))); dst->f_seq =cpu_to_be64 (fid_seq (src)); dst->f_oid =cpu_to_be32 (fid_oid (src)); dst->f_ver =cpu_to_be32 (fid_ver (src));
@@ -869,11 +852,6 @@ static inline void fid_cpu_to_be(struct lu_fid *dst, const struct lu_fid *src)static inline void fid_be_to_cpu (struct lu_fid *dst,const struct lu_fid *src) { -/* check that all fields are converted */ -CLASSERT (sizeof (*src) == -sizeof (fid_seq (src)) + -sizeof (fid_oid (src)) + -sizeof (fid_ver (src))); dst->f_seq =be64_to_cpu (fid_seq (src)); dst->f_oid =be32_to_cpu (fid_oid (src)); dst->f_ver =be32_to_cpu (fid_ver (src));
@@ -897,11 +875,6 @@ extern void lustre_swab_lu_seq_range(struct lu_seq_range *range);static inline int lu_fid_eq (const struct lu_fid *f0,const struct lu_fid *f1) { -/* Check that there is no alignment padding. */ -CLASSERT (sizeof (*f0) == -sizeof (f0->f_seq) + -sizeof (f0->f_oid) + -sizeof (f0->f_ver));return memcmp (f0, f1,sizeof (*f0)) ==0 ; }
@@ -3315,9 +3288,10 @@ struct obdo {#define o_grant_used o_data_versionstatic inline void lustre_set_wire_obdo (struct obd_connect_data *ocd, -struct obdo *wobdo,struct obdo *lobdo) +struct obdo *wobdo, +const struct obdo *lobdo) { -memcpy (wobdo, lobdo,sizeof (*lobdo)); + *wobdo = *lobdo; wobdo->o_flags &= ~OBD_FL_LOCAL_MASK ;if (ocd ==NULL )return ;
@@ -3332,16 +3306,15 @@ static inline void lustre_set_wire_obdo(struct obd_connect_data *ocd, }static inline void lustre_get_wire_obdo (struct obd_connect_data *ocd, -struct obdo *lobdo,struct obdo *wobdo) +struct obdo *lobdo, +const struct obdo *wobdo) {obd_flag local_flags =0 ;if (lobdo->o_valid &OBD_MD_FLFLAGS ) local_flags = lobdo->o_flags &OBD_FL_LOCAL_MASK ; -LASSERT (!(wobdo->o_flags &OBD_FL_LOCAL_MASK )); - -memcpy (lobdo, wobdo,sizeof (*lobdo)); + *lobdo = *wobdo;if (local_flags !=0 ) { lobdo->o_valid |=OBD_MD_FLFLAGS ; lobdo->o_flags &= ~OBD_FL_LOCAL_MASK ;
Signature · 1 line
1.7.9.5
PATCH 29/40 staging/lustre/llite: extended attribute cache
In reply to Peng TaoFrom: Andrew Perepechko <[email protected]>
This patch implements an extended attribute cache for a Lustre client. It is organized as a write-through
cache: reads are performed from cache, updates are sent synchronously to the MDS. An additional inode bit MDS_INODELOCK_XATTR is added to protect the cache.
Lustre-change: http://review.whamcloud.com/5537
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2869
--- .../lustre/lustre/include/linux/lustre_lite.h | 1 + .../lustre/lustre/include/lustre/lustre_idl.h | 12 +- .../lustre/lustre/include/lustre_req_layout.h | 3 + drivers/staging/lustre/lustre/include/md_object.h | 4 +- drivers/staging/lustre/lustre/include/obd.h | 5 + .../staging/lustre/lustre/include/obd_support.h | 1 + drivers/staging/lustre/lustre/ldlm/ldlm_lock.c | 2 + drivers/staging/lustre/lustre/llite/Makefile | 2 +- drivers/staging/lustre/lustre/llite/file.c | 13 +- .../staging/lustre/lustre/llite/llite_internal.h | 36 +- drivers/staging/lustre/lustre/llite/llite_lib.c | 22 +- drivers/staging/lustre/lustre/llite/lproc_llite.c | 36 ++ drivers/staging/lustre/lustre/llite/namei.c | 4 + drivers/staging/lustre/lustre/llite/super25.c | 4 + drivers/staging/lustre/lustre/llite/xattr.c | 102 ++-- drivers/staging/lustre/lustre/llite/xattr_cache.c | 640 ++++++++++++++++++++ drivers/staging/lustre/lustre/mdc/mdc_internal.h | 1 + drivers/staging/lustre/lustre/mdc/mdc_locks.c | 63 +- drivers/staging/lustre/lustre/ptlrpc/layout.c | 34 ++ 19 files changed, 923 insertions(+), 62 deletions(-) create mode 100644 drivers/staging/lustre/lustre/llite/xattr_cache.c
drivers/staging/lustre/lustre/include/linux/lustre_lite.h +1 -0
@@ -88,6 +88,7 @@ enum {LPROC_LL_ALLOC_INODE ,LPROC_LL_SETXATTR ,LPROC_LL_GETXATTR , +LPROC_LL_GETXATTR_HITS ,LPROC_LL_LISTXATTR ,LPROC_LL_REMOVEXATTR , LPROC_LL_INODE_PERM,
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +9 -3
@@ -1357,7 +1357,7 @@ extern void lustre_swab_ptlrpc_body(struct ptlrpc_body *pb); OBD_CONNECT_EINPROGRESS | \ OBD_CONNECT_LIGHTWEIGHT | OBD_CONNECT_UMASK | \ OBD_CONNECT_LVB_TYPE | OBD_CONNECT_LAYOUTLOCK |\ - OBD_CONNECT_PINGLESS) + OBD_CONNECT_PINGLESS | OBD_CONNECT_MAX_EASIZE)#define OST_CONNECT_SUPPORTED (OBD_CONNECT_SRVLOCK | OBD_CONNECT_GRANT | \ OBD_CONNECT_REQPORTAL | OBD_CONNECT_VERSION | \ OBD_CONNECT_TRUNCLOCK | OBD_CONNECT_INDEX | \
@@ -1741,7 +1741,9 @@ static inline __u32 lov_mds_md_size(__u16 stripes, __u32 lmm_magic)#define OBD_MD_FLCKSPLIT (0x0000080000000000ULL)/* Check split on server */ #define OBD_MD_FLCROSSREF (0x0000100000000000ULL)/* Cross-ref case */ #define OBD_MD_FLGETATTRLOCK (0x0000200000000000ULL)/* Get IOEpoch attributes -* under lock */ +* under lock; for xattr +* requests means the +* client holds the lock */ #define OBD_MD_FLOBJCOUNT (0x0000400000000000ULL)/* for multiple destroy */ #define OBD_MD_FLRMTLSETFACL (0x0001000000000000ULL)/* lfs lsetfacl case */
@@ -1758,6 +1760,9 @@ static inline __u32 lov_mds_md_size(__u16 stripes, __u32 lmm_magic) OBD_MD_FLGID | OBD_MD_FLFLAGS | OBD_MD_FLNLINK | \ OBD_MD_FLGENER | OBD_MD_FLRDEV | OBD_MD_FLGROUP) +#define OBD_MD_FLXATTRLOCKED OBD_MD_FLGETATTRLOCK +#define OBD_MD_FLXATTRALL (OBD_MD_FLXATTR | OBD_MD_FLXATTRLS) +/* don't forget obdo_fid which is way down at the bottom so it can * come after the definition of llog_cookie */
@@ -2130,8 +2135,9 @@ extern void lustre_swab_generic_32s (__u32 *val);#define MDS_INODELOCK_OPEN 0x000004/* For opened files */ #define MDS_INODELOCK_LAYOUT 0x000008/* for layout */ #define MDS_INODELOCK_PERM 0x000010/* for permission */ +#define MDS_INODELOCK_XATTR 0x000020/* extended attributes */ -#define MDS_INODELOCK_MAXSHIFT 4 +#define MDS_INODELOCK_MAXSHIFT 5/* This FULL lock is useful to take on unlink sort of operations */ #define MDS_INODELOCK_FULL ((1<<(MDS_INODELOCK_MAXSHIFT+1))-1)
drivers/staging/lustre/lustre/include/lustre_req_layout.h +3 -0
@@ -230,6 +230,7 @@ extern struct req_format RQF_LDLM_INTENT_GETATTR;extern struct req_format RQF_LDLM_INTENT_OPEN ;extern struct req_format RQF_LDLM_INTENT_CREATE ;extern struct req_format RQF_LDLM_INTENT_UNLINK ; +extern struct req_format RQF_LDLM_INTENT_GETXATTR ;extern struct req_format RQF_LDLM_INTENT_QUOTA ;extern struct req_format RQF_LDLM_CANCEL ;extern struct req_format RQF_LDLM_CALLBACK ;
@@ -279,6 +280,8 @@ extern struct req_msg_field RMF_LAYOUT_INTENT;extern struct req_msg_field RMF_MDT_MD ;extern struct req_msg_field RMF_REC_REINT ;extern struct req_msg_field RMF_EADATA ; +extern struct req_msg_field RMF_EAVALS ; +extern struct req_msg_field RMF_EAVALS_LENS ;extern struct req_msg_field RMF_ACL ;extern struct req_msg_field RMF_LOGCOOKIES ;extern struct req_msg_field RMF_CAPA1 ;
drivers/staging/lustre/lustre/include/md_object.h +2 -2
@@ -352,8 +352,8 @@ struct md_device_operations {int (*mdo_root_get)(const struct lu_env *env,struct md_device *m,struct lu_fid *f); -int (*mdo_maxsize_get)(const struct lu_env *env,struct md_device *m, -int *md_size,int *cookie_size); +int (*mdo_maxeasize_get)(const struct lu_env *env,struct md_device *m, +int *easize);int (*mdo_statfs)(const struct lu_env *env,struct md_device *m,struct obd_statfs *sfs);
drivers/staging/lustre/lustre/include/obd.h +5 -0
@@ -1022,6 +1022,7 @@ struct lu_context;#define IT_LAYOUT (1 << 10)#define IT_QUOTA_DQACQ (1 << 11)#define IT_QUOTA_CONN (1 << 12) +#define IT_SETXATTR (1 << 13)static inline int it_to_lock_mode (struct lookup_intent *it) {
@@ -1031,6 +1032,10 @@ static inline int it_to_lock_mode(struct lookup_intent *it)else if (it->it_op & (IT_READDIR |IT_GETATTR |IT_OPEN |IT_LOOKUP |IT_LAYOUT ))return LCK_CR ; +else if (it->it_op &IT_GETXATTR ) +return LCK_PR ; +else if (it->it_op &IT_SETXATTR ) +return LCK_PW ;LASSERTF (0 ,"Invalid it_op: %d\n" , it->it_op);return -EINVAL ;
drivers/staging/lustre/lustre/include/obd_support.h +1 -0
@@ -466,6 +466,7 @@ int obd_alloc_fail(const void *ptr, const char *name, const char *type,#define OBD_FAIL_LOCK_STATE_WAIT_INTR 0x1402#define OBD_FAIL_LOV_INIT 0x1403#define OBD_FAIL_GLIMPSE_DELAY 0x1404 +#define OBD_FAIL_LLITE_XATTR_ENOMEM 0x1405#define OBD_FAIL_FID_INDIR 0x1501#define OBD_FAIL_FID_INLMA 0x1502
drivers/staging/lustre/lustre/ldlm/ldlm_lock.c +2 -0
@@ -145,6 +145,8 @@ char *ldlm_it2str(int it)return "getxattr" ;case IT_LAYOUT :return "layout" ; +case IT_SETXATTR : +return "setxattr" ;default :CERROR ("Unknown intent %d\n" , it);return "UNKNOWN" ;
drivers/staging/lustre/lustre/llite/Makefile +1 -1
@@ -2,7 +2,7 @@ obj-$(CONFIG_LUSTRE_FS) += lustre.o obj-$(CONFIG_LUSTRE_LLITE_LLOOP) += llite_lloop.o lustre-y := dcache.o dir.o file.o llite_close.o llite_lib.o llite_nfs.o \ rw.o lproc_llite.o namei.o symlink.o llite_mmap.o \ - xattr.o remote_perm.o llite_rmtacl.o llite_capa.o \ + xattr.o xattr_cache.o remote_perm.o llite_rmtacl.o llite_capa.o \ rw26.o super25.o statahead.o \ ../lclient/glimpse.o ../lclient/lcommon_cl.o ../lclient/lcommon_misc.o \ vvp_dev.o vvp_page.o vvp_lock.o vvp_io.o vvp_object.o
drivers/staging/lustre/lustre/llite/file.c +8 -5
@@ -2782,7 +2782,8 @@ int ll_have_md_lock(struct inode *inode, __u64 *bits, ldlm_mode_t l_req_mode) }ldlm_mode_t ll_take_md_lock (struct inode *inode,__u64 bits, -struct lustre_handle *lockh,__u64 flags) +struct lustre_handle *lockh,__u64 flags, +ldlm_mode_t mode) {ldlm_policy_data_t policy = { .l_inodebits = {bits}};struct lu_fid *fid;
@@ -2792,8 +2793,8 @@ ldlm_mode_t ll_take_md_lock(struct inode *inode, __u64 bits,CDEBUG (D_INFO ,"trying to match res " DFID "\n" ,PFID (fid)); rc =md_lock_match (ll_i2mdexp (inode),LDLM_FL_BLOCK_GRANTED |flags, - fid,LDLM_IBITS , &policy, -LCK_CR |LCK_CW |LCK_PR |LCK_PW , lockh); + fid,LDLM_IBITS , &policy, mode, lockh); +return rc; }
@@ -3470,7 +3471,8 @@ int ll_layout_refresh(struct inode *inode, __u32 *gen)/* mostly layout lock is caching on the local side, so try to match * it before grabbing layout lock mutex. */ - mode =ll_take_md_lock (inode,MDS_INODELOCK_LAYOUT , &lockh,0 ); + mode =ll_take_md_lock (inode,MDS_INODELOCK_LAYOUT , &lockh,0 , +LCK_CR |LCK_CW |LCK_PR |LCK_PW );if (mode !=0 ) {/* hit cached lock */ rc =ll_layout_lock_set (&lockh, mode, inode, gen, false);if (rc ==0 )
@@ -3485,7 +3487,8 @@ int ll_layout_refresh(struct inode *inode, __u32 *gen) again:/* try again. Maybe somebody else has done this. */ - mode =ll_take_md_lock (inode,MDS_INODELOCK_LAYOUT , &lockh,0 ); + mode =ll_take_md_lock (inode,MDS_INODELOCK_LAYOUT , &lockh,0 , +LCK_CR |LCK_CW |LCK_PR |LCK_PW );if (mode !=0 ) {/* hit cached lock */ rc =ll_layout_lock_set (&lockh, mode, inode, gen, true);if (rc == -EAGAIN )
drivers/staging/lustre/lustre/llite/llite_internal.h +33 -3
@@ -127,6 +127,8 @@ enum lli_flags {LLIF_DATA_MODIFIED = (1 <<6 ),/* File is being restored */ LLIF_FILE_RESTORING = (1 <<7 ), +/* Xattr cache is attached to the file */ + LLIF_XATTR_CACHE = (1 << 8), }; struct ll_inode_info {
@@ -279,8 +281,27 @@ struct ll_inode_info {struct mutex lli_layout_mutex;/* valid only inside LAYOUT ibits lock, protected by lli_layout_mutex */ __u32 lli_layout_gen; + +struct rw_semaphore lli_xattrs_list_rwsem; +struct mutex lli_xattrs_enq_lock; +struct list_head lli_xattrs;/* ll_xattr_entry->xe_list */ }; +int ll_xattr_cache_destroy (struct inode *inode); + +int ll_xattr_cache_get (struct inode *inode, +const char *name, +char *buffer, +size_t size, +__u64 valid); + +int ll_xattr_cache_update (struct inode *inode, +const char *name, +const char *newval, +size_t size, +__u64 valid, +int flags); +/* * Locking to guarantee consistency of non-atomic updates to long long i_size, * consistency between file size and KMS.
@@ -402,6 +423,7 @@ enum stats_track_type {#define LL_SBI_VERBOSE 0x10000/* verbose mount/umount */ #define LL_SBI_LAYOUT_LOCK 0x20000/* layout lock support */ #define LL_SBI_USER_FID2PATH 0x40000/* allow fid2path by unprivileged users */ +#define LL_SBI_XATTR_CACHE 0x80000/* support for xattr cache */ #define LL_SBI_FLAGS { \ "nolck", \
@@ -409,6 +431,7 @@ enum stats_track_type {"flock" , \"xattr" , \"acl" , \ +"???" , \"rmt_client" , \"mds_capa" , \ "oss_capa", \
@@ -421,7 +444,9 @@ enum stats_track_type {"agl" , \"verbose" , \"layout" , \ - "user_fid2path" } +"user_fid2path" ,\ +"xattr" , \ +} /* default value for ll_sb_info->contention_time */ #define SBI_DEFAULT_CONTENTION_SECONDS 60
@@ -461,7 +486,8 @@ struct ll_sb_info {struct lu_fid ll_root_fid;/* root object fid */ int ll_flags; - int ll_umounting:1; + unsigned int ll_umounting:1, + ll_xattr_cache_enabled:1;struct list_head ll_conn_chain;/* per-conn chain of SBs */ struct lustre_client_ocd ll_lco;
@@ -732,7 +758,8 @@ extern int ll_inode_revalidate_it(struct dentry *, struct lookup_intent *,extern int ll_have_md_lock (struct inode *inode,__u64 *bits,ldlm_mode_t l_req_mode);extern ldlm_mode_t ll_take_md_lock (struct inode *inode,__u64 bits, -struct lustre_handle *lockh,__u64 flags); +struct lustre_handle *lockh,__u64 flags, +ldlm_mode_t mode);int __ll_inode_revalidate_it (struct dentry *,struct lookup_intent *,__u64 bits);int ll_revalidate_nd (struct dentry *dentry,unsigned int flags);
@@ -1598,4 +1625,7 @@ int ll_layout_conf(struct inode *inode, const struct cl_object_conf *conf);int ll_layout_refresh (struct inode *inode,__u32 *gen);int ll_layout_restore (struct inode *inode); +int ll_xattr_init (void ); +void ll_xattr_fini (void ); +#endif /* LLITE_INTERNAL_H */
drivers/staging/lustre/lustre/llite/llite_lib.c +20 -2
@@ -209,7 +209,8 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt,OBD_CONNECT_FULL20 |OBD_CONNECT_64BITHASH |OBD_CONNECT_EINPROGRESS |OBD_CONNECT_JOBSTATS |OBD_CONNECT_LVB_TYPE | -OBD_CONNECT_LAYOUTLOCK |OBD_CONNECT_PINGLESS ; +OBD_CONNECT_LAYOUTLOCK |OBD_CONNECT_PINGLESS | +OBD_CONNECT_MAX_EASIZE ;if (sbi->ll_flags &LL_SBI_SOM_PREVIEW ) data->ocd_connect_flags |=OBD_CONNECT_SOM ;
@@ -383,6 +384,16 @@ static int client_common_fill_super(struct super_block *sb, char *md, char *dt, sbi->ll_flags |=LL_SBI_LAYOUT_LOCK ; } +if (data->ocd_ibits_known &MDS_INODELOCK_XATTR ) { +if (!(data->ocd_connect_flags &OBD_CONNECT_MAX_EASIZE )) { +LCONSOLE_INFO ("%s: disabling xattr cache due to " +"unknown maximum xattr size.\n" , dt); + }else { + sbi->ll_flags |=LL_SBI_XATTR_CACHE ; + sbi->ll_xattr_cache_enabled =1 ; + } + } + obd =class_name2obd (dt);if (!obd) {CERROR ("DT %s: not setup or attached\n" , dt);
@@ -922,6 +933,9 @@ void ll_lli_init(struct ll_inode_info *lli) lli->lli_layout_gen =LL_LAYOUT_GEN_NONE ; lli->lli_clob =NULL ; +init_rwsem (&lli->lli_xattrs_list_rwsem); +mutex_init (&lli->lli_xattrs_enq_lock); +LASSERT (lli->lli_vfs_inode.i_mode !=0 );if (S_ISDIR (lli->lli_vfs_inode.i_mode)) {mutex_init (&lli->lli_readdir_mutex);
@@ -1194,6 +1208,8 @@ void ll_clear_inode(struct inode *inode) lli->lli_symlink_name =NULL ; } +ll_xattr_cache_destroy (inode); +if (sbi->ll_flags &LL_SBI_RMT_CLIENT ) {LASSERT (lli->lli_posix_acl ==NULL );if (lli->lli_remote_perms) {
@@ -1753,7 +1769,9 @@ void ll_update_inode(struct inode *inode, struct lustre_md *md)* lock on the client and set LLIF_MDS_SIZE_LOCK holding * it. */ mode =ll_take_md_lock (inode,MDS_INODELOCK_UPDATE , - &lockh,LDLM_FL_CBPENDING ); + &lockh,LDLM_FL_CBPENDING , +LCK_CR |LCK_CW | +LCK_PR |LCK_PW );if (mode) { if (lli->lli_flags & (LLIF_DONE_WRITING | LLIF_EPOCH_PENDING |
drivers/staging/lustre/lustre/llite/lproc_llite.c +36 -0
@@ -723,6 +723,40 @@ static int ll_sbi_flags_seq_show(struct seq_file *m, void *v) }LPROC_SEQ_FOPS_RO (ll_sbi_flags); +static int ll_xattr_cache_seq_show (struct seq_file *m,void *v) +{ +struct super_block *sb = m->private; +struct ll_sb_info *sbi =ll_s2sbi (sb); +int rc; + + rc =seq_printf (m,"%u\n" , sbi->ll_xattr_cache_enabled); + +return rc; +} + +static ssize_t ll_xattr_cache_seq_write (struct file *file,const char *buffer, +size_t count,loff_t *off) +{ +struct super_block *sb = ((struct seq_file *)file->private_data)->private; +struct ll_sb_info *sbi =ll_s2sbi (sb); +int val, rc; + + rc =lprocfs_write_helper (buffer, count, &val); +if (rc) +return rc; + +if (val !=0 && val !=1 ) +return -ERANGE ; + +if (val ==1 && !(sbi->ll_flags &LL_SBI_XATTR_CACHE )) +return -ENOTSUPP ; + + sbi->ll_xattr_cache_enabled = val; + +return count; +} +LPROC_SEQ_FOPS (ll_xattr_cache); +static struct lprocfs_vars lprocfs_llite_obd_vars[] = { {"uuid" , &ll_sb_uuid_fops,0 ,0 },//{ "mntpt_path", ll_rd_path, 0, 0 },
@@ -751,6 +785,7 @@ static struct lprocfs_vars lprocfs_llite_obd_vars[] = { { "lazystatfs", &ll_lazystatfs_fops, 0 }, { "max_easize", &ll_maxea_size_fops, 0, 0 }, { "sbi_flags", &ll_sbi_flags_fops, 0, 0 }, + { "xattr_cache", &ll_xattr_cache_fops, 0, 0 }, { 0 } };
@@ -802,6 +837,7 @@ struct llite_file_opcode { { LPROC_LL_ALLOC_INODE, LPROCFS_TYPE_REGS, "alloc_inode" }, { LPROC_LL_SETXATTR, LPROCFS_TYPE_REGS, "setxattr" }, { LPROC_LL_GETXATTR, LPROCFS_TYPE_REGS, "getxattr" }, + { LPROC_LL_GETXATTR_HITS, LPROCFS_TYPE_REGS, "getxattr_hits" }, { LPROC_LL_LISTXATTR, LPROCFS_TYPE_REGS, "listxattr" }, { LPROC_LL_REMOVEXATTR, LPROCFS_TYPE_REGS, "removexattr" }, { LPROC_LL_INODE_PERM, LPROCFS_TYPE_REGS, "inode_permission" },
drivers/staging/lustre/lustre/llite/namei.c +4 -0
@@ -222,6 +222,10 @@ int ll_md_blocking_ast(struct ldlm_lock *lock, struct ldlm_lock_desc *desc,break ;LASSERT (lock->l_flags &LDLM_FL_CANCELING ); + +if (bits &MDS_INODELOCK_XATTR ) +ll_xattr_cache_destroy (inode); +/* For OPEN locks we differentiate between lock modes * LCK_CR, LCK_CW, LCK_PR - bug 22891 */ if (bits & (MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE |
drivers/staging/lustre/lustre/llite/super25.c +4 -0
@@ -187,11 +187,15 @@ static int __init init_lustre_lite(void)if (rc ==0 ) rc =vvp_global_init (); +if (rc ==0 ) + rc =ll_xattr_init (); +return rc; } static void __exit exit_lustre_lite(void) { +ll_xattr_fini ();vvp_global_fini ();del_timer (&ll_capa_timer);ll_capa_thread_stop ();
drivers/staging/lustre/lustre/llite/xattr.c +57 -45
135 lines not on this page · raw message
drivers/staging/lustre/lustre/llite/xattr_cache.c new file +640 -0
640 lines not on this page · raw message
drivers/staging/lustre/lustre/mdc/mdc_internal.h +1 -0
@@ -72,6 +72,7 @@ void mdc_open_pack(struct ptlrpc_request *req, struct md_op_data *op_data, __u32 mode, __u64 rdev, __u64 flags, const void *data, int datalen);void mdc_unlink_pack (struct ptlrpc_request *req,struct md_op_data *op_data); +void mdc_getxattr_pack (struct ptlrpc_request *req,struct md_op_data *op_data);void mdc_link_pack (struct ptlrpc_request *req,struct md_op_data *op_data);void mdc_rename_pack (struct ptlrpc_request *req,struct md_op_data *op_data,const char *old,int oldlen,const char *new,int newlen);
drivers/staging/lustre/lustre/mdc/mdc_locks.c +62 -1
89 lines not on this page · raw message
drivers/staging/lustre/lustre/ptlrpc/layout.c +34 -0
@@ -462,6 +462,25 @@ static const struct req_msg_field *ldlm_intent_unlink_client[] = { &RMF_NAME }; +static const struct req_msg_field *ldlm_intent_getxattr_client[] = { + &RMF_PTLRPC_BODY , + &RMF_DLM_REQ , + &RMF_LDLM_INTENT , + &RMF_MDT_BODY , + &RMF_CAPA1 , +}; + +static const struct req_msg_field *ldlm_intent_getxattr_server[] = { + &RMF_PTLRPC_BODY , + &RMF_DLM_REP , + &RMF_MDT_BODY , + &RMF_MDT_MD , + &RMF_ACL ,/* for req_capsule_extend/mdt_intent_policy */ + &RMF_EADATA , + &RMF_EAVALS , + &RMF_EAVALS_LENS +}; +static const struct req_msg_field *mds_getxattr_client[] = { &RMF_PTLRPC_BODY , &RMF_MDT_BODY ,
@@ -739,6 +758,7 @@ static struct req_format *req_formats[] = { &RQF_LDLM_INTENT_OPEN , &RQF_LDLM_INTENT_CREATE , &RQF_LDLM_INTENT_UNLINK , + &RQF_LDLM_INTENT_GETXATTR , &RQF_LDLM_INTENT_QUOTA , &RQF_QUOTA_DQACQ , &RQF_LOG_CANCEL,
@@ -1013,6 +1033,9 @@ struct req_msg_field RMF_EADATA = DEFINE_MSGF("eadata", 0, -1, NULL, NULL);EXPORT_SYMBOL (RMF_EADATA ); +struct req_msg_field RMF_EAVALS =DEFINE_MSGF ("eavals" ,0 ,-1 ,NULL ,NULL ); +EXPORT_SYMBOL (RMF_EAVALS ); +struct req_msg_field RMF_ACL =DEFINE_MSGF ("acl" ,RMF_F_NO_SIZE_CHECK ,LUSTRE_POSIX_ACL_MAX_SIZE ,NULL ,NULL );
@@ -1064,6 +1087,11 @@ struct req_msg_field RMF_RCS = lustre_swab_generic_32s, dump_rcs);EXPORT_SYMBOL (RMF_RCS ); +struct req_msg_field RMF_EAVALS_LENS = +DEFINE_MSGF ("eavals_lens" ,RMF_F_STRUCT_ARRAY ,sizeof (__u32), + lustre_swab_generic_32s,NULL ); +EXPORT_SYMBOL (RMF_EAVALS_LENS ); +struct req_msg_field RMF_OBD_ID =DEFINE_MSGF ("obd_id" ,0 ,sizeof (obd_id), lustre_swab_ost_last_id,NULL );
@@ -1421,6 +1449,12 @@ struct req_format RQF_LDLM_INTENT_UNLINK = ldlm_intent_unlink_client, ldlm_intent_server);EXPORT_SYMBOL (RQF_LDLM_INTENT_UNLINK ); +struct req_format RQF_LDLM_INTENT_GETXATTR = +DEFINE_REQ_FMT0 ("LDLM_INTENT_GETXATTR" , + ldlm_intent_getxattr_client, + ldlm_intent_getxattr_server); +EXPORT_SYMBOL (RQF_LDLM_INTENT_GETXATTR ); +struct req_format RQF_MDS_CLOSE =DEFINE_REQ_FMT0 ("MDS_CLOSE" , mdt_close_client, mds_last_unlink_server);
Signature · 1 line
1.7.9.5
PATCH 25/40 staging/lustre/build: clean up unused variables and dead code
In reply to Peng TaoFrom: Dmitry Eremin <[email protected]>
Clean up the code from really unused variables and rearrange other to remove SET_BUT_UNUSED and UNUSED macro usage if possible.
Clean up or comment on some dead code.
Removed never implemented suspend timeouts functionality from pinger.c which was commented out since 2007 and going to be replaced by adaptive timeouts. Also removed all references to this functionality from ldlm_lockd.c, ldlm_request.c and import.c which actually nevers executes or do nothing.
pinger.c commit d2d56f38da01001c92a09afc6b52b5acbd9bc13c
Author: tappro <tappro>
Date: Mon Jul 30 21:08:59 2007 +0000
- make HEAD from b_post_cmd3
Lustre-change: http://review.whamcloud.com/6139
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3204
--- .../staging/lustre/include/linux/libcfs/libcfs.h | 2 - drivers/staging/lustre/lnet/selftest/rpc.c | 2 - drivers/staging/lustre/lnet/selftest/selftest.h | 3 - drivers/staging/lustre/lnet/selftest/timer.c | 6 +- .../lustre/lustre/include/linux/lustre_compat25.h | 4 +- drivers/staging/lustre/lustre/include/lustre_ha.h | 3 - drivers/staging/lustre/lustre/include/lustre_net.h | 2 - drivers/staging/lustre/lustre/ldlm/ldlm_request.c | 3 - drivers/staging/lustre/lustre/libcfs/workitem.c | 55 ++++++++--------- drivers/staging/lustre/lustre/llite/dcache.c | 36 ++++------- drivers/staging/lustre/lustre/llite/namei.c | 6 +- .../lustre/lustre/obdclass/lprocfs_status.c | 4 -- drivers/staging/lustre/lustre/ptlrpc/client.c | 10 ++- drivers/staging/lustre/lustre/ptlrpc/import.c | 2 - drivers/staging/lustre/lustre/ptlrpc/niobuf.c | 10 ++- drivers/staging/lustre/lustre/ptlrpc/pinger.c | 65 -------------------- drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c | 44 ++++++------- 17 files changed, 75 insertions(+), 182 deletions(-)
drivers/staging/lustre/include/linux/libcfs/libcfs.h +0 -2
@@ -181,8 +181,6 @@ static inline void *__container_of(void *ptr, unsigned long shift)#define container_of0 (ptr, type, member) \ ((type *)__container_of((void *)(ptr), offsetof(type, member))) -#define SET_BUT_UNUSED (a) do { } while(sizeof(a) - sizeof(a)) -#define _LIBCFS_H#endif /* _LIBCFS_H */
drivers/staging/lustre/lnet/selftest/rpc.c +0 -2
@@ -124,7 +124,6 @@ srpc_bulk_t * srpc_alloc_bulk(int cpt, unsigned bulk_npg, unsigned bulk_len, int sink) { srpc_bulk_t *bk; -struct page **pages;int i;LASSERT (bulk_npg >0 && bulk_npg <=LNET_MAX_IOV );
@@ -140,7 +139,6 @@ srpc_alloc_bulk(int cpt, unsigned bulk_npg, unsigned bulk_len, int sink) bk->bk_sink = sink; bk->bk_len = bulk_len; bk->bk_niov = bulk_npg; -UNUSED (pages);for (i =0 ; i < bulk_npg; i++) {struct page *pg;
drivers/staging/lustre/lnet/selftest/selftest.h +0 -3
@@ -572,9 +572,6 @@ swi_state2str (int state)#undef STATE2STR } -#define UNUSED (x) ( (void)(x) ) - -#define selftest_wait_events () cfs_pause(cfs_time_seconds(1) / 10)
drivers/staging/lustre/lnet/selftest/timer.c +2 -4
@@ -172,9 +172,6 @@ int stt_timer_main(void *arg) {int rc =0 ; -UNUSED (arg); - -SET_BUT_UNUSED (rc);cfs_block_allsigs ();
@@ -184,12 +181,13 @@ stt_timer_main(void *arg) rc =wait_event_timeout (stt_data.stt_waitq, stt_data.stt_shuttingdown,cfs_time_seconds (STTIMER_SLOTTIME )); + rc =0 ;/* Discard jiffies remaining before timeout. */ }spin_lock (&stt_data.stt_lock); stt_data.stt_nthreads--;spin_unlock (&stt_data.stt_lock); -return 0 ; +return rc; }int
drivers/staging/lustre/lustre/include/linux/lustre_compat25.h +2 -2
@@ -171,8 +171,8 @@ static inline int ll_quota_off(struct super_block *sb, int off, int remount)#define ll_d_hlist_empty (list) hlist_empty(list)#define ll_d_hlist_entry (ptr, type, name) hlist_entry(ptr.first, type, name)#define ll_d_hlist_for_each (tmp, i_dentry) hlist_for_each(tmp, i_dentry) -#define ll_d_hlist_for_each_entry (dentry, p, i_dentry, alias) \ - p = NULL; hlist_for_each_entry(dentry, i_dentry, alias) +#define ll_d_hlist_for_each_entry (dentry, i_dentry, alias) \ + hlist_for_each_entry(dentry, i_dentry, alias)#define bio_hw_segments (q, bio) 0
drivers/staging/lustre/lustre/include/lustre_ha.h +0 -3
@@ -58,9 +58,6 @@ void ptlrpc_activate_import(struct obd_import *imp);void ptlrpc_deactivate_import (struct obd_import *imp);void ptlrpc_invalidate_import (struct obd_import *imp);void ptlrpc_fail_import (struct obd_import *imp,__u32 conn_cnt); -int ptlrpc_check_suspend (void ); -void ptlrpc_activate_timeouts (struct obd_import *imp); -void ptlrpc_deactivate_timeouts (struct obd_import *imp);/** @} ha */
drivers/staging/lustre/lustre/include/lustre_net.h +0 -2
@@ -3403,10 +3403,8 @@ int ptlrpc_del_timeout_client(struct list_head *obd_list, enum timeout_event event);struct ptlrpc_request *ptlrpc_prep_ping (struct obd_import *imp);int ptlrpc_obd_ping (struct obd_device *obd); -cfs_time_t ptlrpc_suspend_wakeup_time (void );void ping_evictor_start (void );void ping_evictor_stop (void ); -int ptlrpc_check_and_wait_suspend (struct ptlrpc_request *req);void ptlrpc_pinger_ir_up (void );void ptlrpc_pinger_ir_down (void );/** @} */
drivers/staging/lustre/lustre/ldlm/ldlm_request.c +0 -3
@@ -97,9 +97,6 @@ int ldlm_expired_completion_wait(void *data)if (lock->l_conn_export ==NULL ) {static cfs_time_t next_dump =0 , last_dump =0 ; -if (ptlrpc_check_suspend ()) -return 0 ; - LCONSOLE_WARN("lock timed out (enqueued at "CFS_TIME_T", " CFS_DURATION_T"s ago)\n", lock->l_last_activity,
drivers/staging/lustre/lustre/libcfs/workitem.c +26 -29
89 lines not on this page · raw message
drivers/staging/lustre/lustre/llite/dcache.c +13 -23
@@ -270,7 +270,6 @@ void ll_intent_release(struct lookup_intent *it)void ll_invalidate_aliases (struct inode *inode) {struct dentry *dentry; -struct ll_d_hlist_node *p;LASSERT (inode !=NULL );
@@ -278,7 +277,7 @@ void ll_invalidate_aliases(struct inode *inode) inode->i_ino, inode->i_generation, inode);ll_lock_dcache (inode); -ll_d_hlist_for_each_entry (dentry, p, &inode->i_dentry, d_alias) { +ll_d_hlist_for_each_entry (dentry, &inode->i_dentry, d_alias) { CDEBUG(D_DENTRY, "dentry in drop %.*s (%p) parent %p " "inode %p flags %d\n", dentry->d_name.len, dentry->d_name.name, dentry, dentry->d_parent,
@@ -404,7 +403,6 @@ int ll_revalidate_it(struct dentry *de, int lookup_flags,struct inode *inode = de->d_inode;struct ll_inode_info *lli =ll_i2info (inode);struct obd_client_handle **och_p; -__u64 *och_usecount;__u64 ibits;/*
@@ -417,38 +415,30 @@ int ll_revalidate_it(struct dentry *de, int lookup_flags,* change, LOOKUP lock is revoked. */ - -if (it->it_flags &FMODE_WRITE ) { +if (it->it_flags &FMODE_WRITE ) och_p = &lli->lli_mds_write_och; - och_usecount = &lli->lli_open_fd_write_count; - }else if (it->it_flags &FMODE_EXEC ) { +else if (it->it_flags &FMODE_EXEC ) och_p = &lli->lli_mds_exec_och; - och_usecount = &lli->lli_open_fd_exec_count; - }else { +else och_p = &lli->lli_mds_read_och; - och_usecount = &lli->lli_open_fd_read_count; - }/* Check for the proper lock. */ ibits =MDS_INODELOCK_LOOKUP ;if (!ll_have_md_lock (inode, &ibits,LCK_MINMODE )) goto do_lock;mutex_lock (&lli->lli_och_mutex);if (*och_p) {/* Everything is open already, do nothing */ -/*(*och_usecount)++; Do not let them steal our open -handle from under us */ -SET_BUT_UNUSED (och_usecount); -/* XXX The code above was my original idea, but in case -we have the handle, but we cannot use it due to later -checks (e.g. O_CREAT|O_EXCL flags set), nobody -would decrement counter increased here. So we just -hope the lock won't be invalidated in between. But -if it would be, we'll reopen the open request to -MDS later during file open path */ +/* Originally it was idea to do not let them steal our +* open handle from under us by (*och_usecount)++ here. +* But in case we have the handle, but we cannot use it +* due to later checks (e.g. O_CREAT|O_EXCL flags set), +* nobody would decrement counter increased here. So we +* just hope the lock won't be invalidated in between. +* But if it would be, we'll reopen the open request to +* MDS later during file open path. */ mutex_unlock (&lli->lli_och_mutex);return 1 ; - }else { -mutex_unlock (&lli->lli_och_mutex); } +mutex_unlock (&lli->lli_och_mutex); }if (it->it_op ==IT_GETATTR ) {
drivers/staging/lustre/lustre/llite/namei.c +2 -4
@@ -172,10 +172,9 @@ struct inode *ll_iget(struct super_block *sb, ino_t hash,static void ll_invalidate_negative_children (struct inode *dir) {struct dentry *dentry, *tmp_subdir; -struct ll_d_hlist_node *p;ll_lock_dcache (dir); -ll_d_hlist_for_each_entry (dentry, p, &dir->i_dentry, d_alias) { +ll_d_hlist_for_each_entry (dentry, &dir->i_dentry, d_alias) {spin_lock (&dentry->d_lock);if (!list_empty (&dentry->d_subdirs)) {struct dentry *child;
@@ -352,7 +351,6 @@ void ll_i2gids(__u32 *suppgids, struct inode *i1, struct inode *i2)static struct dentry *ll_find_alias (struct inode *inode,struct dentry *dentry) {struct dentry *alias, *discon_alias, *invalid_alias; -struct ll_d_hlist_node *p;if (ll_d_hlist_empty (&inode->i_dentry))return NULL ;
@@ -360,7 +358,7 @@ static struct dentry *ll_find_alias(struct inode *inode, struct dentry *dentry) discon_alias = invalid_alias =NULL ;ll_lock_dcache (inode); -ll_d_hlist_for_each_entry (alias, p, &inode->i_dentry, d_alias) { +ll_d_hlist_for_each_entry (alias, &inode->i_dentry, d_alias) {LASSERT (alias != dentry);spin_lock (&alias->d_lock);
drivers/staging/lustre/lustre/obdclass/lprocfs_status.c +0 -4
@@ -420,7 +420,6 @@ void lprocfs_stats_collect(struct lprocfs_stats *stats, int idx, {unsigned int num_entry;struct lprocfs_counter *percpu_cntr; -struct lprocfs_counter_header *cntr_header;int i;unsigned long flags =0 ;
@@ -439,7 +438,6 @@ void lprocfs_stats_collect(struct lprocfs_stats *stats, int idx,for (i =0 ; i < num_entry; i++) {if (stats->ls_percpu[i] ==NULL )continue ; - cntr_header = &stats->ls_cnt_header[idx]; percpu_cntr =lprocfs_stats_counter_get (stats, i, idx); cnt->lc_count += percpu_cntr->lc_count;
@@ -999,7 +997,6 @@ EXPORT_SYMBOL(lprocfs_free_stats);void lprocfs_clear_stats (struct lprocfs_stats *stats) {struct lprocfs_counter *percpu_cntr; -struct lprocfs_counter_header *header;int i;int j;unsigned int num_entry;
@@ -1011,7 +1008,6 @@ void lprocfs_clear_stats(struct lprocfs_stats *stats)if (stats->ls_percpu[i] ==NULL )continue ;for (j =0 ; j < stats->ls_num; j++) { - header = &stats->ls_cnt_header[j]; percpu_cntr =lprocfs_stats_counter_get (stats, i, j); percpu_cntr->lc_count =0 ; percpu_cntr->lc_min =LC_MIN_INIT ;
drivers/staging/lustre/lustre/ptlrpc/client.c +4 -6
@@ -2288,7 +2288,6 @@ EXPORT_SYMBOL(ptlrpc_req_xid);int ptlrpc_unregister_reply (struct ptlrpc_request *request,int async) {int rc; -wait_queue_head_t *wq;struct l_wait_info lwi;/*
@@ -2333,12 +2332,11 @@ int ptlrpc_unregister_reply(struct ptlrpc_request *request, int async)* a chance to run reply_in_callback(), and to make sure we've * unlinked before returning a req to the pool. */ -if (request->rq_set !=NULL ) - wq = &request->rq_set->set_waitq; -else - wq = &request->rq_reply_waitq; -for (;;) { +/* The wq argument is ignored by user-space wait_event macros */ +wait_queue_head_t *wq = (request->rq_set !=NULL ) ? + &request->rq_set->set_waitq : + &request->rq_reply_waitq;/* Network access will complete in finite time but the HUGE * timeout lets us CWARN for visibility of sluggish NALs */ lwi = LWI_TIMEOUT_INTERVAL(cfs_time_seconds(LONG_UNLINK),
drivers/staging/lustre/lustre/ptlrpc/import.c +0 -2
@@ -170,7 +170,6 @@ int ptlrpc_set_import_discon(struct obd_import *imp, __u32 conn_cnt) target_len, target_start, libcfs_nid2str(imp->imp_connection->c_peer.nid)); } -ptlrpc_deactivate_timeouts (imp);IMPORT_SET_STATE_NOLOCK (imp,LUSTRE_IMP_DISCON );spin_unlock (&imp->imp_lock);
@@ -383,7 +382,6 @@ void ptlrpc_activate_import(struct obd_import *imp)spin_lock (&imp->imp_lock); imp->imp_invalid =0 ; -ptlrpc_activate_timeouts (imp);spin_unlock (&imp->imp_lock);obd_import_event (obd, imp,IMP_EVENT_ACTIVE ); }
drivers/staging/lustre/lustre/ptlrpc/niobuf.c +4 -6
@@ -242,7 +242,6 @@ EXPORT_SYMBOL(ptlrpc_register_bulk);int ptlrpc_unregister_bulk (struct ptlrpc_request *req,int async) {struct ptlrpc_bulk_desc *desc = req->rq_bulk; -wait_queue_head_t *wq;struct l_wait_info lwi;int rc;
@@ -274,12 +273,11 @@ int ptlrpc_unregister_bulk(struct ptlrpc_request *req, int async)if (async)return 0 ; -if (req->rq_set !=NULL ) - wq = &req->rq_set->set_waitq; -else - wq = &req->rq_reply_waitq; -for (;;) { +/* The wq argument is ignored by user-space wait_event macros */ +wait_queue_head_t *wq = (req->rq_set !=NULL ) ? + &req->rq_set->set_waitq : + &req->rq_reply_waitq;/* Network access will complete in finite time but the HUGE * timeout lets us CWARN for visibility of sluggish NALs */ lwi = LWI_TIMEOUT_INTERVAL(cfs_time_seconds(LONG_UNLINK),
drivers/staging/lustre/lustre/ptlrpc/pinger.c +0 -65
83 lines not on this page · raw message
drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c +22 -22
75 lines not on this page · raw message
Signature · 1 line
1.7.9.5
PATCH 24/40 staging/lustre/llite: don't check for O_CREAT in it_create_mode
In reply to Peng TaoFrom: "John L. Hammond" <[email protected]>
ll_lookup_it() checks for O_CREAT in struct lookup_intent's it_create_mode member which is nonsensical, as it_create_mode is used for file mode bits (S_IFREG, S_IRUSR, ...). Fix this by just checking for IT_CREATE in it_op and do the same in llu_lookup_it(). This will not affect the behavior of either function, since if O_CREATE (0100) is actually set in o_create_mode then IT_CREATE must have been set in it_op. In ll_atomic_open() check for O_CREAT in the open_flags parameter rather than testing mode.
Lustre-change: http://review.whamcloud.com/6786
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3517
--- drivers/staging/lustre/lustre/llite/namei.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/staging/lustre/lustre/llite/namei.c +2 -3
@@ -523,8 +523,7 @@ static struct dentry *ll_lookup_it(struct inode *parent, struct dentry *dentry, icbd.icbd_childp = &dentry; icbd.icbd_parent = parent; -if (it->it_op &IT_CREAT || - (it->it_op &IT_OPEN && it->it_create_mode &O_CREAT )) +if (it->it_op &IT_CREAT ) opc =LUSTRE_OPC_CREATE ;else opc =LUSTRE_OPC_ANY ;
@@ -623,7 +622,7 @@ static int ll_atomic_open(struct inode *dir, struct dentry *dentry,return -ENOMEM ; it->it_op =IT_OPEN ; -if (mode) { +if (open_flags &O_CREAT ) { it->it_op |=IT_CREAT ; lookup_flags |=LOOKUP_CREATE ; }
Signature · 1 line
1.7.9.5
PATCH 23/40 staging/lustre/scrub: control OI scrub on OST from user space
In reply to Peng TaoFrom: Fan Yong <[email protected]>
Not all the OI inconsistency can be detected automatically, such as /O entry lost case. So the OI scrub on OST should can be triggered by the administrator manually.
Lustre-change: http://review.whamcloud.com/6698
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3335
--- .../staging/lustre/lustre/include/obd_support.h | 1 + 1 file changed, 1 insertion(+)
drivers/staging/lustre/lustre/include/obd_support.h +1 -0
@@ -257,6 +257,7 @@ int obd_alloc_fail(const void *ptr, const char *name, const char *type,#define OBD_FAIL_OSD_FID_MAPPING 0x193#define OBD_FAIL_OSD_LMA_INCOMPAT 0x194#define OBD_FAIL_OSD_COMPAT_INVALID_ENTRY 0x195 +#define OBD_FAIL_OSD_COMPAT_NO_ENTRY 0x196#define OBD_FAIL_OST 0x200#define OBD_FAIL_OST_CONNECT_NET 0x201
Signature · 1 line
1.7.9.5
PATCH 22/40 staging/lustre/scrub: OI scrub on OST
In reply to Peng TaoFrom: Fan Yong <[email protected]>
OI scrub should has the ability to handle kinds of OI, including both the OI files on MDT and the /O directory on OST.
We trust the FID in LMA for both MDT objects and OST objects. So if some /O sub-item does not match related LMA, then the /O will be updated, instead of the LMA.
To guarantee that the OI scrub can run without MDT0 involved for FLDB, the OST object needs to store some flag in its LMA to tell OI scrub that it is for an OST object, no need to query the MDT0.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3335
Lustre-change: http://review.whamcloud.com/6669
--- .../lustre/lustre/include/lustre/lustre_idl.h | 18 +++++++++++------- .../staging/lustre/lustre/include/obd_support.h | 1 + drivers/staging/lustre/lustre/obdclass/md_attrs.c | 4 ++-- drivers/staging/lustre/lustre/ptlrpc/wiretest.c | 4 ++++ 4 files changed, 18 insertions(+), 9 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +11 -7
@@ -321,8 +321,11 @@ static inline int range_compare_loc(const struct lu_seq_range *r1,* xattr. */ enum lma_compat { -LMAC_HSM =0x00000001 , -LMAC_SOM =0x00000002 , +LMAC_HSM =0x00000001 , +LMAC_SOM =0x00000002 , +LMAC_NOT_IN_OI =0x00000004 ,/* the object does NOT need OI mapping */ +LMAC_FID_ON_OST =0x00000008 ,/* For OST-object, its OI mapping is +* under /O/<seq>/d<x>. */ };/**
@@ -331,16 +334,17 @@ enum lma_compat {* This information is stored in lustre_mdt_attrs::lma_incompat. */ enum lma_incompat { -LMAI_RELEASED =0x0000001 ,/* file is released */ -LMAI_AGENT =0x00000002 ,/* agent inode */ -LMAI_REMOTE_PARENT =0x00000004 ,/* the parent of the object -is on the remote MDT */ +LMAI_RELEASED =0x00000001 ,/* file is released */ +LMAI_AGENT =0x00000002 ,/* agent inode */ +LMAI_REMOTE_PARENT =0x00000004 ,/* the parent of the object +is on the remote MDT */ };#define LMA_INCOMPAT_SUPP (LMAI_AGENT | LMAI_REMOTE_PARENT)extern void lustre_lma_swab (struct lustre_mdt_attrs *lma);extern void lustre_lma_init (struct lustre_mdt_attrs *lma, -const struct lu_fid *fid,__u32 incompat); +const struct lu_fid *fid, +__u32 compat,__u32 incompat);/** * SOM on-disk attributes stored in a separate xattr. */
drivers/staging/lustre/lustre/include/obd_support.h +1 -0
@@ -256,6 +256,7 @@ int obd_alloc_fail(const void *ptr, const char *name, const char *type,#define OBD_FAIL_OSD_SCRUB_FATAL 0x192#define OBD_FAIL_OSD_FID_MAPPING 0x193#define OBD_FAIL_OSD_LMA_INCOMPAT 0x194 +#define OBD_FAIL_OSD_COMPAT_INVALID_ENTRY 0x195#define OBD_FAIL_OST 0x200#define OBD_FAIL_OST_CONNECT_NET 0x201
drivers/staging/lustre/lustre/obdclass/md_attrs.c +2 -2
@@ -39,9 +39,9 @@* \param incompat - features that MDS must understand to access object */ void lustre_lma_init (struct lustre_mdt_attrs *lma,const struct lu_fid *fid, -__u32 incompat) +__u32 compat,__u32 incompat) { - lma->lma_compat =0 ; + lma->lma_compat = compat; lma->lma_incompat = incompat; lma->lma_self_fid = *fid;
drivers/staging/lustre/lustre/ptlrpc/wiretest.c +4 -0
@@ -432,6 +432,10 @@ void lustre_assert_wire_constants(void) (unsigned)LMAC_HSM);LASSERTF (LMAC_SOM ==0x00000002UL ,"found 0x%.8xUL\n" , (unsigned )LMAC_SOM ); +LASSERTF (LMAC_NOT_IN_OI ==0x00000004UL ,"found 0x%.8xUL\n" , + (unsigned )LMAC_NOT_IN_OI ); +LASSERTF (LMAC_FID_ON_OST ==0x00000008UL ,"found 0x%.8xUL\n" , + (unsigned )LMAC_FID_ON_OST );LASSERTF (OBJ_CREATE ==1 ,"found %lld\n" , (long long )OBJ_CREATE ); LASSERTF(OBJ_DESTROY == 2, "found %lld\n",
Signature · 1 line
1.7.9.5
Re: [PATCH 12/40] staging/lustre/nfs: writing to new files will return ENOENT
In reply to Peng TaoHi Peng,
This patch was eventually reverted as it was causing interoperability issues with 2.1 Lustre server. We are working on the new patch, so please don't include this one.
Thanks,
- Cheng
Quoted 61 lines · Peng Tao
From: Patrick Farrell <[email protected]>
This happend with SLES11SP2 Lustre client, which in turn acts as an NFS server, exporting a subtree of an Lustre fs through NFS.
We detected that whenever we are writing to a new file using, fx, 'echo blah > newfile', it will return ENOENT error. We found out that this was caused by the anonymous dentry. In SLESS11SP2, anonymous dentries are assigned '/' as the name, instead of an empty string. When MDT handles the
intent_opencall, it will look up the obj by the name if it is not an empty string, and thus couldn't find it.As
MDS_OPEN_BY_FIDis always set on this request, we never need to send the name in this request. The fid is already available and should be used in case the file has been renamed.Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3544
Lustre-change: http://review.whamcloud.com/6920Signed-off-by Cheng Shao <[email protected]>Signed-off-by Patrick Farrell <[email protected]>Reviewed-by Bob Glossman <[email protected]>Reviewed-by Alexey Shvetsov <[email protected]>Reviewed-by Lai Siyao <[email protected]>Reviewed-by James Simmons <[email protected]>Reviewed-by Oleg Drokin <[email protected]>Signed-off-by Peng Tao <[email protected]>Signed-off-by Andreas Dilger <[email protected]>---
drivers/staging/lustre/lustre/llite/file.c| 5 ++---1 file changed, 2 insertions(+), 3 deletions(-)drivers/staging/lustre/lustre/llite/file.c +2
-3@@ -368,8 +368,6 @@ static int ll_intent_file_open(struct file *file, void *lmm, {struct ll_sb_info *sbi =ll_i2sbi (file->f_dentry->d_inode);struct dentry *parent = file->f_dentry->d_parent; -const char *name = file->f_dentry->d_name.name; -const int len = file->f_dentry->d_name.len;struct md_op_data *op_data;struct ptlrpc_request *req;__u32 opc =LUSTRE_OPC_ANY ;@@ -394,8 +392,9 @@ static int ll_intent_file_open(struct file *file, void *lmm, } op_data =ll_prep_md_op_data (NULL , parent->d_inode, - file->f_dentry->d_inode, name, len, + file->f_dentry->d_inode,NULL ,0 ,O_RDWR , opc,NULL ); +if (IS_ERR (op_data))return PTR_ERR (op_data);Signature · 1 line
1.7.9.5
PATCH 20/40 staging/lustre/mdt: HSM coordinator client interface
In reply to Peng TaoFrom: JC Lafoucriere <[email protected]>
This patch implements the HSM coordinator methods
used by client to add/remove/list HSM actions on
FID.
Lustre-change: http://review.whamcloud.com/6532
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3341
--- .../lustre/lustre/include/lustre/lustre_user.h | 7 +++---- drivers/staging/lustre/lustre/mdc/mdc_request.c | 4 +--- 2 files changed, 4 insertions(+), 7 deletions(-)
drivers/staging/lustre/lustre/include/lustre/lustre_user.h +3 -4
@@ -1121,11 +1121,10 @@ static inline int hal_size(struct hsm_action_list *hal) sz =sizeof (*hal) +cfs_size_round (strlen (hal->hal_fsname)); hai =hai_zero (hal); -for (i =0 ; i < hal->hal_count ; i++) { +for (i =0 ; i < hal->hal_count ; i++, hai =hai_next (hai)) sz +=cfs_size_round (hai->hai_len); - hai =hai_next (hai); - } -return (sz); + +return sz; }/* Copytool progress reporting */
drivers/staging/lustre/lustre/mdc/mdc_request.c +1 -3
@@ -1919,10 +1919,8 @@ static void lustre_swab_hal(struct hsm_action_list *h)__swab32s (&h->hal_archive_id);__swab64s (&h->hal_flags); hai =hai_zero (h); -for (i =0 ; i < h->hal_count; i++) { +for (i =0 ; i < h->hal_count; i++, hai =hai_next (hai))lustre_swab_hai (hai); - hai =hai_next (hai); - } }static void lustre_swab_kuch (struct kuc_hdr *l)
Signature · 1 line
1.7.9.5
PATCH 18/40 staging/lustre/mdc: prevent fall through in mdc_iocontrol()
In reply to Peng TaoFrom: "John L. Hammond" <[email protected]>
In mdc_iocontrol() add a goto to the end of the LL_IOC_HSM_STATE_SET case, preventing fall through into the next case. In the same function, replace the return statement in OBD_IOC_QUOTACTL with a goto, so that a reference to the module is not leaked.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3576
Lustre-change: http://review.whamcloud.com/6962
--- drivers/staging/lustre/lustre/mdc/mdc_request.c | 27 +++++++++++------------ 1 file changed, 13 insertions(+), 14 deletions(-)
drivers/staging/lustre/lustre/mdc/mdc_request.c +13 -14
@@ -1743,6 +1743,7 @@ static int mdc_iocontrol(unsigned int cmd, struct obd_export *exp, int len,GOTO (out, rc);case LL_IOC_HSM_STATE_SET : rc =mdc_ioc_hsm_state_set (exp, karg); +GOTO (out, rc);case LL_IOC_HSM_ACTION : rc =mdc_ioc_hsm_current_action (exp, karg);GOTO (out, rc);
@@ -1814,8 +1815,8 @@ static int mdc_iocontrol(unsigned int cmd, struct obd_export *exp, int len,struct obd_quotactl *oqctl;OBD_ALLOC_PTR (oqctl); -if (!oqctl) -return -ENOMEM ; +if (oqctl ==NULL ) +GOTO (out, rc = -ENOMEM );QCTL_COPY (oqctl, qctl); rc =obd_quotactl (exp, oqctl);
@@ -1824,23 +1825,21 @@ static int mdc_iocontrol(unsigned int cmd, struct obd_export *exp, int len, qctl->qc_valid =QC_MDTIDX ; qctl->obd_uuid = obd->u.cli.cl_target_uuid; } +OBD_FREE_PTR (oqctl); -break ; +GOTO (out, rc); } -case LL_IOC_GET_CONNECT_FLAGS : { -if (copy_to_user (uarg, -exp_connect_flags_ptr (exp), -sizeof (__u64))) +case LL_IOC_GET_CONNECT_FLAGS : +if (copy_to_user (uarg,exp_connect_flags_ptr (exp), +sizeof (*exp_connect_flags_ptr (exp))))GOTO (out, rc = -EFAULT ); -else -GOTO (out, rc =0 ); - } -case LL_IOC_LOV_SWAP_LAYOUTS : { + +GOTO (out, rc =0 ); +case LL_IOC_LOV_SWAP_LAYOUTS : rc =mdc_ioc_swap_layouts (exp, karg); -break ; - } +GOTO (out, rc);default : -CERROR ("mdc_ioctl(): unrecognised ioctl %#x\n" , cmd); +CERROR ("unrecognised ioctl: cmd = %#x\n" , cmd);GOTO (out, rc = -ENOTTY ); } out:
Signature · 1 line
1.7.9.5
PATCH 17/40 staging/lustre/ptlrpc: Fix race during exp_flock_hash creation
In reply to Peng TaoFrom: Andriy Skulysh <[email protected]>
During race exp_flock_hash can be created 2 times. It is created & assigned without any lock.
Move hash initialization from ldlm_flock_blocking_link() to ldlm_init_export()
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2835
Lustre-change: http://review.whamcloud.com/5471
--- drivers/staging/lustre/lustre/ldlm/ldlm_flock.c | 28 +++++++---------------- drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 8 +++++++ 2 files changed, 16 insertions(+), 20 deletions(-)
drivers/staging/lustre/lustre/ldlm/ldlm_flock.c +8 -20
@@ -95,20 +95,12 @@ ldlm_flocks_overlap(struct ldlm_lock *lock, struct ldlm_lock *new) lock->l_policy_data.l_flock.start)); } -static inline int ldlm_flock_blocking_link (struct ldlm_lock *req, -struct ldlm_lock *lock) +static inline void ldlm_flock_blocking_link (struct ldlm_lock *req, +struct ldlm_lock *lock) { -int rc =0 ; -/* For server only */ if (req->l_export ==NULL ) -return 0 ; - -if (unlikely (req->l_export->exp_flock_hash ==NULL )) { - rc =ldlm_init_flock_export (req->l_export); -if (rc) - goto error; - } +return ;LASSERT (hlist_unhashed (&req->l_exp_flock_hash));
@@ -121,8 +113,6 @@ static inline int ldlm_flock_blocking_link(struct ldlm_lock *req,cfs_hash_add (req->l_export->exp_flock_hash, &req->l_policy_data.l_flock.owner, &req->l_exp_flock_hash); -error: -return rc; }static inline void ldlm_flock_blocking_unlink (struct ldlm_lock *req)
@@ -250,7 +240,6 @@ ldlm_process_flock_lock(struct ldlm_lock *req, __u64 *flags, int first_enq,int overlaps =0 ;int splitted =0 ;const struct ldlm_callback_suite null_cbs = {NULL }; -int rc; CDEBUG(D_DLMTRACE, "flags %#llx owner "LPU64" pid %u mode %u start " LPU64" end "LPU64"\n", *flags,
@@ -328,12 +317,8 @@ reprocess:/* add lock to blocking list before deadlock * check to prevent race */ - rc =ldlm_flock_blocking_link (req, lock); -if (rc) { -ldlm_flock_destroy (req, mode, *flags); - *err = rc; -return LDLM_ITER_STOP ; - } +ldlm_flock_blocking_link (req, lock); +if (ldlm_flock_deadlock (req, lock)) {ldlm_flock_blocking_unlink (req);ldlm_flock_destroy (req, mode, *flags);
@@ -816,6 +801,9 @@ static cfs_hash_ops_t ldlm_export_flock_ops = {int ldlm_init_flock_export (struct obd_export *exp) { +if (strcmp (exp->exp_obd->obd_type->typ_name,LUSTRE_MDT_NAME ) !=0 ) +return 0 ; + exp->exp_flock_hash = cfs_hash_create(obd_uuid2str(&exp->exp_client_uuid), HASH_EXP_LOCK_CUR_BITS,
drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c +8 -0
@@ -964,6 +964,7 @@ static cfs_hash_ops_t ldlm_export_lock_ops = {int ldlm_init_export (struct obd_export *exp) { +int rc; exp->exp_lock_hash = cfs_hash_create(obd_uuid2str(&exp->exp_client_uuid), HASH_EXP_LOCK_CUR_BITS,
@@ -977,7 +978,14 @@ int ldlm_init_export(struct obd_export *exp)if (!exp->exp_lock_hash)return -ENOMEM ; + rc =ldlm_init_flock_export (exp); +if (rc) +GOTO (err, rc); +return 0 ; +err: +ldlm_destroy_export (exp); +return rc; }EXPORT_SYMBOL (ldlm_init_export);
Signature · 1 line
1.7.9.5
PATCH 12/40 staging/lustre/nfs: writing to new files will return ENOENT
In reply to Peng TaoFrom: Patrick Farrell <[email protected]>
This happend with SLES11SP2 Lustre client, which in turn acts as an NFS server, exporting a subtree of an Lustre fs through NFS.
We detected that whenever we are writing to a new file using, fx, 'echo blah > newfile', it will return ENOENT error. We found out that this was caused by the anonymous dentry. In SLESS11SP2, anonymous dentries are assigned '/' as the name, instead of an empty string. When MDT handles the intent_open call, it will look up the obj by the name if it is not an empty string, and thus couldn't find it.
As MDS_OPEN_BY_FID is always set on this request, we never need to send the name in this request. The fid is already available and should be used in case the file has been renamed.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3544
Lustre-change: http://review.whamcloud.com/6920
--- drivers/staging/lustre/lustre/llite/file.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/staging/lustre/lustre/llite/file.c +2 -3
@@ -368,8 +368,6 @@ static int ll_intent_file_open(struct file *file, void *lmm, {struct ll_sb_info *sbi =ll_i2sbi (file->f_dentry->d_inode);struct dentry *parent = file->f_dentry->d_parent; -const char *name = file->f_dentry->d_name.name; -const int len = file->f_dentry->d_name.len;struct md_op_data *op_data;struct ptlrpc_request *req;__u32 opc =LUSTRE_OPC_ANY ;
@@ -394,8 +392,9 @@ static int ll_intent_file_open(struct file *file, void *lmm, } op_data =ll_prep_md_op_data (NULL , parent->d_inode, - file->f_dentry->d_inode, name, len, + file->f_dentry->d_inode,NULL ,0 ,O_RDWR , opc,NULL ); +if (IS_ERR (op_data))return PTR_ERR (op_data);
Signature · 1 line
1.7.9.5
PATCH 11/40 staging/lustre/llog: MGC to use OSD API for backup logs
In reply to Peng TaoFrom: Mikhail Pershin <[email protected]>
MGC uses lvfs API to access local llogs blocking removal of old code
- MGS is converted to use OSD API for local llogs
- llog_is_empty() and llog_backup() are introduced
- initial OSD start initialize run ctxt so llog can use it too
- shrink dcache after initial scrub in osd-ldiskfs to avoid holding data of unlinked objects until umount.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2059
Lustre-change: http://review.whamcloud.com/5049
--- drivers/staging/lustre/lustre/include/dt_object.h | 2 +- drivers/staging/lustre/lustre/include/lustre_log.h | 13 +- drivers/staging/lustre/lustre/include/obd.h | 4 +- drivers/staging/lustre/lustre/mgc/libmgc.c | 3 - drivers/staging/lustre/lustre/mgc/mgc_request.c | 401 ++++++++++++-------- drivers/staging/lustre/lustre/obdclass/llog.c | 212 +++++++---- .../staging/lustre/lustre/obdclass/local_storage.c | 9 +- .../staging/lustre/lustre/obdclass/local_storage.h | 3 + drivers/staging/lustre/lustre/obdclass/lu_object.c | 2 +- drivers/staging/lustre/lustre/obdclass/obd_mount.c | 3 + 10 files changed, 408 insertions(+), 244 deletions(-)
drivers/staging/lustre/lustre/include/dt_object.h +1 -1
@@ -692,7 +692,7 @@ struct local_oid_storage {struct dt_object *los_obj;/* data used to generate new fids */ -struct mutex los_id_lock; +struct mutex los_id_lock;__u64 los_seq;__u32 los_last_oid; };
drivers/staging/lustre/lustre/include/lustre_log.h +12 -1
@@ -136,7 +136,11 @@ int llog_open(const struct lu_env *env, struct llog_ctxt *ctxt, struct llog_handle **lgh, struct llog_logid *logid, char *name, enum llog_open_param open_param);int llog_close (const struct lu_env *env,struct llog_handle *cathandle); -int llog_get_size (struct llog_handle *loghandle); +int llog_is_empty (const struct lu_env *env,struct llog_ctxt *ctxt, +char *name); +int llog_backup (const struct lu_env *env,struct obd_device *obd, +struct llog_ctxt *ctxt,struct llog_ctxt *bak_ctxt, +char *name,char *backup);/* llog_process flags */ #define LLOG_FLAG_NODEAMON 0x0001
@@ -382,6 +386,13 @@ static inline int llog_data_len(int len)return cfs_size_round (len); } +static inline int llog_get_size (struct llog_handle *loghandle) +{ +if (loghandle && loghandle->lgh_hdr) +return loghandle->lgh_hdr->llh_count; +return 0 ; +} +static inline struct llog_ctxt *llog_ctxt_get (struct llog_ctxt *ctxt) {atomic_inc (&ctxt->loc_refcount);
drivers/staging/lustre/lustre/include/obd.h +2 -2
@@ -399,8 +399,8 @@ struct client_obd {/* mgc datastruct */ struct semaphore cl_mgc_sem; -struct vfsmount *cl_mgc_vfsmnt; -struct dentry *cl_mgc_configs_dir; +struct local_oid_storage *cl_mgc_los; +struct dt_object *cl_mgc_configs_dir;atomic_t cl_mgc_refcount;struct obd_export *cl_mgc_mgsexp;
drivers/staging/lustre/lustre/mgc/libmgc.c +0 -3
@@ -99,11 +99,8 @@ static int mgc_precleanup(struct obd_device *obd, enum obd_cleanup_stage stage)static int mgc_cleanup (struct obd_device *obd) { -struct client_obd *cli = &obd->u.cli;int rc; -LASSERT (cli->cl_mgc_vfsmnt ==NULL ); -ptlrpcd_decref (); rc =client_obd_cleanup (obd);
drivers/staging/lustre/lustre/mgc/mgc_request.c +246 -155
527 lines not on this page · raw message
drivers/staging/lustre/lustre/obdclass/llog.c +134 -78
258 lines not on this page · raw message
drivers/staging/lustre/lustre/obdclass/local_storage.c +6 -3
@@ -855,9 +855,12 @@ out_los: (*los)->los_seq =fid_seq (first_fid); (*los)->los_last_oid =le64_to_cpu (lastid); (*los)->los_obj = o; -/* read value should not be less than initial one */ -LASSERTF ((*los)->los_last_oid >= first_oid,"%u < %u\n" , - (*los)->los_last_oid, first_oid); +/* Read value should not be less than initial one +* but possible after upgrade from older fs. +* In this case just switch to the first_oid in memory and +* it will be updated on disk with first object generated */ +if ((*los)->los_last_oid < first_oid) + (*los)->los_last_oid = first_oid; } out:mutex_unlock (&ls->ls_los_mutex);
drivers/staging/lustre/lustre/obdclass/local_storage.h +3 -0
@@ -29,6 +29,8 @@* * Author: Mikhail Pershin <[email protected]> */ +#ifndef __LOCAL_STORAGE_H +#define __LOCAL_STORAGE_H#include <dt_object.h> #include <obd.h>
@@ -86,3 +88,4 @@ struct los_ondisk { };#define LOS_MAGIC 0xdecafbee +#endif
drivers/staging/lustre/lustre/obdclass/lu_object.c +1 -1
@@ -423,7 +423,7 @@ LU_KEY_INIT_FINI(lu_global, struct lu_cdebug_data);*/ struct lu_context_key lu_global_key = { .lct_tags =LCT_MD_THREAD |LCT_DT_THREAD | -LCT_MG_THREAD |LCT_CL_THREAD , +LCT_MG_THREAD |LCT_CL_THREAD |LCT_LOCAL , .lct_init = lu_global_key_init, .lct_fini = lu_global_key_fini };
drivers/staging/lustre/lustre/obdclass/obd_mount.c +3 -0
@@ -631,6 +631,9 @@ int lustre_put_lsi(struct super_block *sb)CDEBUG (D_MOUNT ,"put %p %d\n" , sb,atomic_read (&lsi->lsi_mounts));if (atomic_dec_and_test (&lsi->lsi_mounts)) {if (IS_SERVER (lsi) && lsi->lsi_osd_exp) { +lu_device_put (&lsi->lsi_dt_dev->dd_lu_dev); + lsi->lsi_osd_exp->exp_obd->obd_lvfs_ctxt.dt =NULL ; + lsi->lsi_dt_dev =NULL ;obd_disconnect (lsi->lsi_osd_exp);/* wait till OSD is gone */ obd_zombie_barrier ();
Signature · 1 line
1.7.9.5
PATCH 10/40 staging/lustre/server: use unified request handler for MGS
In reply to Peng TaoFrom: Mikhail Pershin <[email protected]>
- Unify request handler. It finds target for particular request and calls appropriate handler for it. Generic handlers are moved to the unified target code. The tgt_session_info is introduced to store all request-related data and passed to all handlers.
- Pack reply in llog server functions early and use err_serious()
- remove obsoleted llog_origin_handle_cancel(), it is not used anymore
- remove push_ctxt/pop_ctxt from llog server function, it is based on OSD now.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2145
Lustre-change: http://review.whamcloud.com/4826
--- .../lustre/lustre/include/lustre_req_layout.h | 2 ++ .../staging/lustre/lustre/include/obd_support.h | 7 +++++++ drivers/staging/lustre/lustre/ptlrpc/layout.c | 7 ++++++- .../staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c | 4 ++-- 4 files changed, 17 insertions(+), 3 deletions(-)
drivers/staging/lustre/lustre/include/lustre_req_layout.h +2 -0
@@ -245,6 +245,8 @@ extern struct req_format RQF_LLOG_ORIGIN_HANDLE_PREV_BLOCK;extern struct req_format RQF_LLOG_ORIGIN_HANDLE_READ_HEADER ;extern struct req_format RQF_LLOG_ORIGIN_CONNECT ; +extern struct req_format RQF_CONNECT ; +extern struct req_msg_field RMF_GENERIC_DATA ;extern struct req_msg_field RMF_PTLRPC_BODY ;extern struct req_msg_field RMF_MDT_BODY ;
drivers/staging/lustre/lustre/include/obd_support.h +7 -0
@@ -416,6 +416,13 @@ int obd_alloc_fail(const void *ptr, const char *name, const char *type,#define OBD_FAIL_MGC_PAUSE_PROCESS_LOG 0x903#define OBD_FAIL_MGS_PAUSE_REQ 0x904#define OBD_FAIL_MGS_PAUSE_TARGET_REG 0x905 +#define OBD_FAIL_MGS_CONNECT_NET 0x906 +#define OBD_FAIL_MGS_DISCONNECT_NET 0x907 +#define OBD_FAIL_MGS_SET_INFO_NET 0x908 +#define OBD_FAIL_MGS_EXCEPTION_NET 0x909 +#define OBD_FAIL_MGS_TARGET_REG_NET 0x90a +#define OBD_FAIL_MGS_TARGET_DEL_NET 0x90b +#define OBD_FAIL_MGS_CONFIG_READ_NET 0x90c#define OBD_FAIL_QUOTA_DQACQ_NET 0xA01#define OBD_FAIL_QUOTA_EDQUOT 0xA02
drivers/staging/lustre/lustre/ptlrpc/layout.c +6 -1
@@ -738,7 +738,8 @@ static struct req_format *req_formats[] = { &RQF_LLOG_ORIGIN_HANDLE_NEXT_BLOCK , &RQF_LLOG_ORIGIN_HANDLE_PREV_BLOCK , &RQF_LLOG_ORIGIN_HANDLE_READ_HEADER , - &RQF_LLOG_ORIGIN_CONNECT + &RQF_LLOG_ORIGIN_CONNECT , + &RQF_CONNECT, }; struct req_msg_field {
@@ -1504,6 +1505,10 @@ struct req_format RQF_LLOG_ORIGIN_CONNECT =DEFINE_REQ_FMT0 ("LLOG_ORIGIN_CONNECT" , llogd_conn_body_only, empty);EXPORT_SYMBOL (RQF_LLOG_ORIGIN_CONNECT ); +struct req_format RQF_CONNECT = +DEFINE_REQ_FMT0 ("CONNECT" , obd_connect_client, obd_connect_server); +EXPORT_SYMBOL (RQF_CONNECT ); +struct req_format RQF_OST_CONNECT =DEFINE_REQ_FMT0 ("OST_CONNECT" , obd_connect_client, obd_connect_server);
drivers/staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c +2 -2
@@ -114,10 +114,10 @@ struct ll_rpc_opcode { { MGS_SET_INFO, "mgs_set_info" }, { MGS_CONFIG_READ, "mgs_config_read" }, { OBD_PING, "obd_ping" }, - { OBD_LOG_CANCEL, "llog_origin_handle_cancel" }, + { OBD_LOG_CANCEL, "llog_cancel" }, { OBD_QC_CALLBACK, "obd_quota_callback" }, { OBD_IDX_READ, "dt_index_read" }, - { LLOG_ORIGIN_HANDLE_CREATE, "llog_origin_handle_create" }, + { LLOG_ORIGIN_HANDLE_CREATE, "llog_origin_handle_open" }, { LLOG_ORIGIN_HANDLE_NEXT_BLOCK, "llog_origin_handle_next_block" }, { LLOG_ORIGIN_HANDLE_READ_HEADER,"llog_origin_handle_read_header" }, { LLOG_ORIGIN_HANDLE_WRITE_REC, "llog_origin_handle_write_rec" },
Signature · 1 line
1.7.9.5
PATCH 9/40 staging/lustre/ldlm: fix resource/fid check, use DLDLMRES
In reply to Peng TaoFrom: Andreas Dilger <[email protected]>
In ll_md_blocking_ast() the FID/resource comparison is incorrectly checking for fid_ver() stored in res_id.name[2] instead of name[1] changed since http://review.whamcloud.com/2271 (commit 4f91d5161d00) landed. This does not impact current clients, since name[2] and fid_ver() are always zero, but it could cause problems in the future.
In ldlm_cli_enqueue_fini() use ldlm_res_eq() instead of comparing each of the resource fields separately.
Use DLDLMRES/PLDLMRES when printing resource names everywhere.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2901
Lustre-change: http://review.whamcloud.com/6592
--- drivers/staging/lustre/lustre/ldlm/ldlm_request.c | 32 +++++++------------- drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 19 +++--------- drivers/staging/lustre/lustre/llite/namei.c | 5 +-- drivers/staging/lustre/lustre/mdc/mdc_locks.c | 9 ++---- drivers/staging/lustre/lustre/mgc/mgc_request.c | 4 +-- 5 files changed, 21 insertions(+), 48 deletions(-)
drivers/staging/lustre/lustre/ldlm/ldlm_request.c +11 -21
@@ -610,18 +610,12 @@ int ldlm_cli_enqueue_fini(struct obd_export *exp, struct ptlrpc_request *req, lock->l_req_mode = newmode; } -if (memcmp (reply->lock_desc.l_resource.lr_name.name, - lock->l_resource->lr_name.name, -sizeof (struct ldlm_res_id ))) { -CDEBUG (D_INFO ,"remote intent success, locking " -"(%ld,%ld,%ld) instead of " -"(%ld,%ld,%ld)\n" , - (long )reply->lock_desc.l_resource.lr_name.name[0 ], - (long )reply->lock_desc.l_resource.lr_name.name[1 ], - (long )reply->lock_desc.l_resource.lr_name.name[2 ], - (long )lock->l_resource->lr_name.name[0 ], - (long )lock->l_resource->lr_name.name[1 ], - (long )lock->l_resource->lr_name.name[2 ]); +if (!ldlm_res_eq (&reply->lock_desc.l_resource.lr_name, + &lock->l_resource->lr_name)) { +CDEBUG (D_INFO ,"remote intent success, locking " DLDLMRES +" instead of " DLDLMRES "\n" , +PLDLMRES (&reply->lock_desc.l_resource), +PLDLMRES (lock->l_resource)); rc =ldlm_lock_change_resource (ns, lock, &reply->lock_desc.l_resource.lr_name);
@@ -1912,7 +1906,8 @@ int ldlm_cli_cancel_unused_resource(struct ldlm_namespace *ns, 0, flags | LCF_BL_AST, opaque); rc =ldlm_cli_cancel_list (&cancels, count,NULL , flags);if (rc !=ELDLM_OK ) -CERROR ("ldlm_cli_cancel_unused_resource: %d\n" , rc); +CERROR ("canceling unused lock " DLDLMRES ": rc = %d\n" , +PLDLMRES (res), rc);LDLM_RESOURCE_DELREF (res);ldlm_resource_putref (res);
@@ -1930,15 +1925,10 @@ static int ldlm_cli_hash_cancel_unused(struct cfs_hash *hs, struct cfs_hash_bd * {struct ldlm_resource *res =cfs_hash_object (hs, hnode);struct ldlm_cli_cancel_arg *lc = arg; -int rc; - rc =ldlm_cli_cancel_unused_resource (ldlm_res_to_ns (res), &res->lr_name, -NULL ,LCK_MINMODE , - lc->lc_flags, lc->lc_opaque); -if (rc !=0 ) { -CERROR ("ldlm_cli_cancel_unused (" LPU64 "): %d\n" , - res->lr_name.name[0 ], rc); - } +ldlm_cli_cancel_unused_resource (ldlm_res_to_ns (res), &res->lr_name, +NULL ,LCK_MINMODE , + lc->lc_flags, lc->lc_opaque);/* must return 0 for hash iteration */ return 0 ; }
drivers/staging/lustre/lustre/ldlm/ldlm_resource.c +5 -14
@@ -762,16 +762,9 @@ static int ldlm_resource_complain(struct cfs_hash *hs, struct cfs_hash_bd *bd,struct ldlm_resource *res =cfs_hash_object (hs, hnode);lock_res (res); -CERROR ("Namespace %s resource refcount nonzero " -"(%d) after lock cleanup; forcing " -"cleanup.\n" , -ldlm_ns_name (ldlm_res_to_ns (res)), -atomic_read (&res->lr_refcount) -1 ); - -CERROR ("Resource: %p (" LPU64 "/" LPU64 "/" LPU64 "/" -LPU64 ") (rc: %d)\n" , res, - res->lr_name.name[0 ], res->lr_name.name[1 ], - res->lr_name.name[2 ], res->lr_name.name[3 ], +CERROR ("%s: namespace resource " DLDLMRES " (%p) refcount nonzero " +"(%d) after lock cleanup; forcing cleanup.\n" , +ldlm_ns_name (ldlm_res_to_ns (res)),PLDLMRES (res), res,atomic_read (&res->lr_refcount) -1 );ldlm_resource_dump (D_ERROR , res);
@@ -1403,10 +1396,8 @@ void ldlm_resource_dump(int level, struct ldlm_resource *res)if (!((libcfs_debug |D_ERROR ) & level))return ; -CDEBUG (level,"--- Resource: %p (" LPU64 "/" LPU64 "/" LPU64 "/" LPU64 -") (rc: %d)\n" , res, res->lr_name.name[0 ], res->lr_name.name[1 ], - res->lr_name.name[2 ], res->lr_name.name[3 ], -atomic_read (&res->lr_refcount)); +CDEBUG (level,"--- Resource: " DLDLMRES " (%p) refcount = %d\n" , +PLDLMRES (res), res,atomic_read (&res->lr_refcount));if (!list_empty (&res->lr_granted)) {CDEBUG (level,"Granted locks (in reverse order):\n" );
drivers/staging/lustre/lustre/llite/namei.c +1 -4
@@ -233,12 +233,9 @@ int ll_md_blocking_ast(struct ldlm_lock *lock, struct ldlm_lock_desc *desc,ll_have_md_lock (inode, &bits, mode); fid =ll_inode2fid (inode); -if (lock->l_resource->lr_name.name[0 ] !=fid_seq (fid) || - lock->l_resource->lr_name.name[1 ] !=fid_oid (fid) || - lock->l_resource->lr_name.name[2 ] !=fid_ver (fid)) { +if (!fid_res_name_eq (fid, &lock->l_resource->lr_name))LDLM_ERROR (lock,"data mismatch with object " DFID " (%p)" ,PFID (fid), inode); - }if (bits &MDS_INODELOCK_OPEN ) {int flags =0 ;
drivers/staging/lustre/lustre/mdc/mdc_locks.c +2 -7
@@ -971,13 +971,8 @@ static int mdc_finish_intent_lock(struct obd_export *exp,LASSERTF (fid_res_name_eq (&mdt_body->fid1, &lock->l_resource->lr_name), -"Lock res_id: %lu/%lu/%lu, fid: %lu/%lu/%lu.\n" , - (unsigned long )lock->l_resource->lr_name.name[0 ], - (unsigned long )lock->l_resource->lr_name.name[1 ], - (unsigned long )lock->l_resource->lr_name.name[2 ], - (unsigned long )fid_seq (&mdt_body->fid1), - (unsigned long )fid_oid (&mdt_body->fid1), - (unsigned long )fid_ver (&mdt_body->fid1)); +"Lock res_id: " DLDLMRES ", fid: " DFID "\n" , +PLDLMRES (lock->l_resource),PFID (&mdt_body->fid1));LDLM_LOCK_PUT (lock);memcpy (&old_lock, lockh,sizeof (*lockh));
drivers/staging/lustre/lustre/mgc/mgc_request.c +2 -2
@@ -788,8 +788,8 @@ static int mgc_blocking_ast(struct ldlm_lock *lock, struct ldlm_lock_desc *desc,/* We've given up the lock, prepare ourselves to update. */ LDLM_DEBUG (lock,"MGC cancel CB" ); -CDEBUG (D_MGC ,"Lock res " LPX64 " (%.8s)\n" , - lock->l_resource->lr_name.name[0 ], +CDEBUG (D_MGC ,"Lock res " DLDLMRES " (%.8s)\n" , +PLDLMRES (lock->l_resource), (char *)&lock->l_resource->lr_name.name[0 ]);if (!cld) {
Signature · 1 line
1.7.9.5
PATCH 8/40 staging/lustre/lnet: Fix assert on empty group in selftest module
In reply to Peng TaoFrom: Amir Shehata <[email protected]>
The core of the issue is that the selftest module doesn't sanitize its own API, but it depends on lst utility to do such checks. As a result this issue manifests itself in this particular LU through an assert on an empty group. If the NID is misspelled then an empty group is added. An error output is provided, but if that's never checked in a batch script, as is the case with this issue, then the script will try to add an empty group to a test to run in a batch, and that will cause an assert
The fix is two fold. Ensure that lst utility checks that a group is added with at least one node. If not the group is subsequently deleted. And the add_test command would fail, since the group no longer exists.
The second fix is to ensure that the kernel module itself sanitizes its own API in this particular case, so that if a different utility is used other than lst to communicate with the selftest kernel module then this error would be caught. This fix looks up the batch and the groups, src and dst, in the ioctl handle and sanitizes that input at this point. If the group looked up either doesn't exist or doesn't have at least one ACTIVE node, then the command fails.
NOTE:there are many other cases in the code where the selftest kernel module doesn't check for sanity of the input, but depends totally on the lst module to do such checks. Particularly around length of strings passed in. Thus it is possible to crash the selftest module if someone tries to create another userspace app to communicate with the selftest kernel module without ensuring sanity of the params sent to the kernel module. In effect, it's always assumed that lst is the front end for selftest and no other front end is to be used.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3093
Lustre-change: http://review.whamcloud.com/6092
--- drivers/staging/lustre/lnet/selftest/conctl.c | 56 ++++++------- drivers/staging/lustre/lnet/selftest/conrpc.c | 2 +- drivers/staging/lustre/lnet/selftest/console.c | 105 ++++++++++++++++-------- drivers/staging/lustre/lnet/selftest/console.h | 6 +- 4 files changed, 102 insertions(+), 67 deletions(-)
drivers/staging/lustre/lnet/selftest/conctl.c +27 -29
@@ -723,12 +723,12 @@ lst_stat_query_ioctl(lstio_stat_args_t *args)int lst_test_add_ioctl (lstio_test_args_t *args) { -char *name; -char *srcgrp =NULL ; -char *dstgrp =NULL ; -void *param =NULL ; -int ret =0 ; -int rc = -ENOMEM ; +char *batch_name; +char *src_name =NULL ; +char *dst_name =NULL ; +void *param =NULL ; +int ret =0 ; +int rc = -ENOMEM ; if (args->lstio_tes_resultp == NULL || args->lstio_tes_retp == NULL ||
@@ -755,16 +755,16 @@ int lst_test_add_ioctl(lstio_test_args_t *args) args->lstio_tes_param_len > PAGE_CACHE_SIZE - sizeof(lstcon_test_t)))return -EINVAL ; -LIBCFS_ALLOC (name, args->lstio_tes_bat_nmlen +1 ); -if (name ==NULL ) +LIBCFS_ALLOC (batch_name, args->lstio_tes_bat_nmlen +1 ); +if (batch_name ==NULL )return rc; -LIBCFS_ALLOC (srcgrp, args->lstio_tes_sgrp_nmlen +1 ); -if (srcgrp ==NULL ) +LIBCFS_ALLOC (src_name, args->lstio_tes_sgrp_nmlen +1 ); +if (src_name ==NULL ) goto out; -LIBCFS_ALLOC (dstgrp, args->lstio_tes_dgrp_nmlen +1 ); -if (dstgrp ==NULL ) +LIBCFS_ALLOC (dst_name, args->lstio_tes_dgrp_nmlen +1 ); +if (dst_name ==NULL ) goto out;if (args->lstio_tes_param !=NULL ) {
@@ -774,39 +774,37 @@ int lst_test_add_ioctl(lstio_test_args_t *args) } rc = -EFAULT ; -if (copy_from_user (name, - args->lstio_tes_bat_name, - args->lstio_tes_bat_nmlen) || -copy_from_user (srcgrp, - args->lstio_tes_sgrp_name, - args->lstio_tes_sgrp_nmlen) || -copy_from_user (dstgrp, - args->lstio_tes_dgrp_name, - args->lstio_tes_dgrp_nmlen) || +if (copy_from_user (batch_name, args->lstio_tes_bat_name, + args->lstio_tes_bat_nmlen) || +copy_from_user (src_name, args->lstio_tes_sgrp_name, + args->lstio_tes_sgrp_nmlen) || +copy_from_user (dst_name, args->lstio_tes_dgrp_name, + args->lstio_tes_dgrp_nmlen) ||copy_from_user (param, args->lstio_tes_param, args->lstio_tes_param_len)) goto out; - rc =lstcon_test_add (name, + rc =lstcon_test_add (batch_name, args->lstio_tes_type, args->lstio_tes_loop, args->lstio_tes_concur, args->lstio_tes_dist, args->lstio_tes_span, - srcgrp, dstgrp, param, args->lstio_tes_param_len, + src_name, dst_name, param, + args->lstio_tes_param_len, &ret, args->lstio_tes_resultp);if (ret !=0 ) rc = (copy_to_user (args->lstio_tes_retp, &ret,sizeof (ret))) ? -EFAULT :0 ; out: -if (name !=NULL ) -LIBCFS_FREE (name, args->lstio_tes_bat_nmlen +1 ); +if (batch_name !=NULL ) +LIBCFS_FREE (batch_name, args->lstio_tes_bat_nmlen +1 ); -if (srcgrp !=NULL ) -LIBCFS_FREE (srcgrp, args->lstio_tes_sgrp_nmlen +1 ); +if (src_name !=NULL ) +LIBCFS_FREE (src_name, args->lstio_tes_sgrp_nmlen +1 ); -if (dstgrp !=NULL ) -LIBCFS_FREE (dstgrp, args->lstio_tes_dgrp_nmlen +1 ); +if (dst_name !=NULL ) +LIBCFS_FREE (dst_name, args->lstio_tes_dgrp_nmlen +1 );if (param !=NULL )LIBCFS_FREE (param, args->lstio_tes_param_len);
drivers/staging/lustre/lnet/selftest/conrpc.c +1 -1
@@ -311,7 +311,7 @@ lstcon_rpc_trans_abort(lstcon_rpc_trans_t *trans, int error)sfw_abort_rpc (rpc); -if (error !=ETIMEDOUT ) +if (error != -ETIMEDOUT )continue ; nd = crpc->crp_node;
drivers/staging/lustre/lnet/selftest/console.c +71 -34
156 lines not on this page · raw message
drivers/staging/lustre/lnet/selftest/console.h +3 -3
@@ -224,9 +224,9 @@ extern int lstcon_group_stat(char *grp_name, int timeout, struct list_head *result_up);extern int lstcon_nodes_stat (int count,lnet_process_id_t *ids_up,int timeout,struct list_head *result_up); -extern int lstcon_test_add (char *name,int type,int loop,int concur, -int dist,int span,char *src_name,char * dst_name, +extern int lstcon_test_add (char *batch_name,int type,int loop, +int concur,int dist,int span, +char *src_name,char *dst_name,void *param,int paramlen,int *retp,struct list_head *result_up); -#endif
Signature · 1 line
1.7.9.5
PATCH 7/40 staging/lustre/hsm: Implementation of exclusive open
In reply to Peng TaoFrom: Jinshan Xiong <[email protected]>
In this patch, a framework of lease is implemented. However, only exclusive lease is supported right now.
To apply a lease, MDS_OPEN_LEASE must be set to open the file, EX mode open lock is returned to the client side to hold a lease. From that time on, if this file is opened again by other processes, the open lock will be revoked so the client who holds the lease will know the lease is already broken by checking that open lock.
To release a lease, normal close is used. The client will revoke the open lock before sending CLOSE request.
Lease can be applied in two ways. ll_lease_open()/close() can be called directly if the lease holder is in kernel space; or if the lease holder lives in user space, it has to open the file first and then use ioctl() with command LL_IOC_SET_LEASE to apply a lease. The lease holder has to poll the lease status itself.
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-2919
Lustre-change: http://review.whamcloud.com/6730
--- .../lustre/lustre/include/linux/lustre_intent.h | 2 +- .../lustre/lustre/include/lustre/lustre_idl.h | 5 + .../lustre/lustre/include/lustre/lustre_user.h | 13 +- .../lustre/lustre/include/lustre_dlm_flags.h | 12 +- drivers/staging/lustre/lustre/include/lustre_lib.h | 11 +- drivers/staging/lustre/lustre/ldlm/ldlm_lock.c | 5 + drivers/staging/lustre/lustre/ldlm/ldlm_request.c | 4 +- drivers/staging/lustre/lustre/llite/file.c | 296 +++++++++++++++++++- .../staging/lustre/lustre/llite/llite_internal.h | 11 +- drivers/staging/lustre/lustre/mdc/mdc_internal.h | 2 +- drivers/staging/lustre/lustre/mdc/mdc_lib.c | 7 +- drivers/staging/lustre/lustre/mdc/mdc_locks.c | 39 ++- 12 files changed, 373 insertions(+), 34 deletions(-)
drivers/staging/lustre/lustre/include/linux/lustre_intent.h +1 -1
@@ -52,8 +52,8 @@ struct lustre_intent_data {struct lookup_intent {int it_op; -int it_flags;int it_create_mode; +__u64 it_flags;union {struct lustre_intent_data lustre; } d;
drivers/staging/lustre/lustre/include/lustre/lustre_idl.h +5 -0
@@ -2117,6 +2117,7 @@ extern void lustre_swab_generic_32s (__u32 *val);#define DISP_ENQ_OPEN_REF 0x00800000#define DISP_ENQ_CREATE_REF 0x01000000#define DISP_OPEN_LOCK 0x02000000 +#define DISP_OPEN_LEASE 0x04000000/* INODE LOCK PARTS */ #define MDS_INODELOCK_LOOKUP 0x000001/* dentry, mode, owner, group */
@@ -2377,6 +2378,10 @@ extern void lustre_swab_mdt_rec_setattr (struct mdt_rec_setattr *sa);* hsm restore) */ #define MDS_OPEN_VOLATILE 0400000000000ULL/* File is volatile = created unlinked */ +#define MDS_OPEN_LEASE 01000000000000ULL/* Open the file and grant lease +* delegation, succeed if it's not +* being opened with conflict mode. +*/ /* permission for create non-directory file */ #define MAY_CREATE (1 << 7)
drivers/staging/lustre/lustre/include/lustre/lustre_user.h +8 -5
@@ -241,12 +241,15 @@ struct ost_id { struct hsm_current_action)/* see <lustre_lib.h> for ioctl numbers 221-232 */ -#define LL_IOC_LMV_SETSTRIPE _IOWR('f', 240, struct lmv_user_md) -#define LL_IOC_LMV_GETSTRIPE _IOWR('f', 241, struct lmv_user_md) -#define LL_IOC_REMOVE_ENTRY _IOWR('f', 242, __u64) +#define LL_IOC_LMV_SETSTRIPE _IOWR('f', 240, struct lmv_user_md) +#define LL_IOC_LMV_GETSTRIPE _IOWR('f', 241, struct lmv_user_md) +#define LL_IOC_REMOVE_ENTRY _IOWR('f', 242, __u64) -#define LL_STATFS_LMV 1 -#define LL_STATFS_LOV 2 +#define LL_IOC_SET_LEASE _IOWR('f', 243, long) +#define LL_IOC_GET_LEASE _IO('f', 244) + +#define LL_STATFS_LMV 1 +#define LL_STATFS_LOV 2#define LL_STATFS_NODELAY 4#define IOC_MDC_TYPE 'i'
drivers/staging/lustre/lustre/include/lustre_dlm_flags.h +10 -2
@@ -35,7 +35,7 @@#ifndef LDLM_ALL_FLAGS_MASK /** l_flags bits marked as "all_flags" bits */ -#define LDLM_FL_ALL_FLAGS_MASK 0x007FFFFFC08F132FULL +#define LDLM_FL_ALL_FLAGS_MASK 0x00FFFFFFC08F132FULL/** l_flags bits marked as "ast" bits */ #define LDLM_FL_AST_MASK 0x0000000080000000ULL
@@ -53,7 +53,7 @@#define LDLM_FL_INHERIT_MASK 0x0000000000800000ULL/** l_flags bits marked as "local_only" bits */ -#define LDLM_FL_LOCAL_ONLY_MASK 0x007FFFFF00000000ULL +#define LDLM_FL_LOCAL_ONLY_MASK 0x00FFFFFF00000000ULL/** l_flags bits marked as "on_wire" bits */ #define LDLM_FL_ON_WIRE_MASK 0x00000000C08F132FULL
@@ -358,6 +358,12 @@#define ldlm_set_ns_srv (_l) LDLM_SET_FLAG(( _l), 1ULL << 54)#define ldlm_clear_ns_srv (_l) LDLM_CLEAR_FLAG((_l), 1ULL << 54) +/** Flag whether this lock can be reused. Used by exclusive open. */ +#define LDLM_FL_EXCL 0x0080000000000000ULL // bit 55 +#define ldlm_is_excl (_l) LDLM_TEST_FLAG(( _l), 1ULL << 55) +#define ldlm_set_excl (_l) LDLM_SET_FLAG(( _l), 1ULL << 55) +#define ldlm_clear_excl (_l) LDLM_CLEAR_FLAG((_l), 1ULL << 55) +/** test for ldlm_lock flag bit set */ #define LDLM_TEST_FLAG (_l, _b) (((_l)->l_flags & (_b)) != 0)
@@ -414,6 +420,7 @@ static int hf_lustre_ldlm_fl_server_lock = -1;static int hf_lustre_ldlm_fl_res_locked =-1 ;static int hf_lustre_ldlm_fl_waited =-1 ;static int hf_lustre_ldlm_fl_ns_srv =-1 ; +static int hf_lustre_ldlm_fl_excl =-1 ;const value_string lustre_ldlm_flags_vals[] = { {LDLM_FL_LOCK_CHANGED ,"LDLM_FL_LOCK_CHANGED" },
@@ -454,6 +461,7 @@ const value_string lustre_ldlm_flags_vals[] = { {LDLM_FL_RES_LOCKED, "LDLM_FL_RES_LOCKED"}, {LDLM_FL_WAITED, "LDLM_FL_WAITED"}, {LDLM_FL_NS_SRV, "LDLM_FL_NS_SRV"}, + {LDLM_FL_EXCL, "LDLM_FL_EXCL"}, { 0, NULL } }; #endif /* WIRESHARK_COMPILE */
drivers/staging/lustre/lustre/include/lustre_lib.h +6 -5
@@ -81,11 +81,12 @@ struct client_obd *client_conn2cli(struct lustre_handle *conn);struct md_open_data ;struct obd_client_handle { -struct lustre_handle och_fh; -struct lu_fid och_fid; -struct md_open_data *och_mod; -__u32 och_magic; -int och_flags; +struct lustre_handle och_fh; +struct lu_fid och_fid; +struct md_open_data *och_mod; +struct lustre_handle och_lease_handle;/* open lock for lease */ +__u32 och_magic; +int och_flags; };#define OBD_CLIENT_HANDLE_MAGIC 0xd15ea5ed
drivers/staging/lustre/lustre/ldlm/ldlm_lock.c +5 -0
@@ -1129,6 +1129,11 @@ static struct ldlm_lock *search_queue(struct list_head *queue,if (lock == old_lock)break ; +/* Check if this lock can be matched. +* Used by LU-2919(exclusive open) for open lease lock */ +if (ldlm_is_excl (lock)) +continue ; +/* llite sometimes wants to match locks that will be * canceled when their users drop, but we allow it to match * if it passes in CBPENDING and the lock still has users.
drivers/staging/lustre/lustre/ldlm/ldlm_request.c +2 -2
@@ -910,7 +910,7 @@ int ldlm_cli_enqueue(struct obd_export *exp, struct ptlrpc_request **reqp, lock->l_conn_export = exp; lock->l_export =NULL ; lock->l_blocking_ast = einfo->ei_cb_bl; - lock->l_flags |= (*flags &LDLM_FL_NO_LRU ); + lock->l_flags |= (*flags & (LDLM_FL_NO_LRU |LDLM_FL_EXCL ));/* lock not sent to server yet */
@@ -1333,7 +1333,7 @@ int ldlm_cli_cancel(struct lustre_handle *lockh, } rc =ldlm_cli_cancel_local (lock); -if (rc ==LDLM_FL_LOCAL_ONLY ) { +if (rc ==LDLM_FL_LOCAL_ONLY || cancel_flags &LCF_LOCAL ) {LDLM_LOCK_RELEASE (lock);return 0 ; }
drivers/staging/lustre/lustre/llite/file.c +295 -1
332 lines not on this page · raw message
drivers/staging/lustre/lustre/llite/llite_internal.h +10 -1
@@ -609,10 +609,14 @@ extern struct kmem_cache *ll_file_data_slab;struct lustre_handle ;struct ll_file_data {struct ll_readahead_state fd_ras; -int fd_omode;struct ccc_grouplock fd_grouplock;__u64 lfd_pos;__u32 fd_flags; +fmode_t fd_omode; +/* openhandle if lease exists for this file. +* Borrow lli->lli_och_mutex to protect assignment */ +struct obd_client_handle *fd_lease_och; +struct obd_client_handle *fd_och;struct file *fd_file;/* Indicate whether need to report failure when close. * true: failure is known, not report again.
@@ -778,6 +782,11 @@ int ll_put_grouplock(struct inode *inode, struct file *file, unsigned long arg);int ll_fid2path (struct inode *inode,void *arg);int ll_data_version (struct inode *inode,__u64 *data_version,int extent_lock); +struct obd_client_handle *ll_lease_open (struct inode *inode,struct file *file, +fmode_t mode); +int ll_lease_close (struct obd_client_handle *och,struct inode *inode, +bool *lease_broken); +/* llite/dcache.c */ int ll_dops_init (struct dentry *de,int block,int init_sa);
drivers/staging/lustre/lustre/mdc/mdc_internal.h +1 -1
@@ -69,7 +69,7 @@ void mdc_create_pack(struct ptlrpc_request *req, struct md_op_data *op_data, const void *data, int datalen, __u32 mode, __u32 uid, __u32 gid, cfs_cap_t capability, __u64 rdev);void mdc_open_pack (struct ptlrpc_request *req,struct md_op_data *op_data, -__u32 mode,__u64 rdev,__u32 flags,const void *data, +__u32 mode,__u64 rdev,__u64 flags,const void *data,int datalen);void mdc_unlink_pack (struct ptlrpc_request *req,struct md_op_data *op_data);void mdc_link_pack (struct ptlrpc_request *req,struct md_op_data *op_data);
drivers/staging/lustre/lustre/mdc/mdc_lib.c +4 -3
@@ -174,12 +174,12 @@ void mdc_create_pack(struct ptlrpc_request *req, struct md_op_data *op_data, } } -static __u64 mds_pack_open_flags (__u32 flags,__u32 mode) +static __u64 mds_pack_open_flags (__u64 flags,__u32 mode) {__u64 cr_flags = (flags & (FMODE_READ |FMODE_WRITE |MDS_OPEN_HAS_EA |MDS_OPEN_HAS_OBJS |MDS_OPEN_OWNEROVERRIDE |MDS_OPEN_LOCK | -MDS_OPEN_BY_FID )); +MDS_OPEN_BY_FID |MDS_OPEN_LEASE ));if (flags &O_CREAT ) cr_flags |=MDS_OPEN_CREAT ;if (flags &O_EXCL )
@@ -207,7 +207,7 @@ static __u64 mds_pack_open_flags(__u32 flags, __u32 mode)/* packing of MDS records */ void mdc_open_pack (struct ptlrpc_request *req,struct md_op_data *op_data, -__u32 mode,__u64 rdev,__u32 flags,const void *lmm, +__u32 mode,__u64 rdev,__u64 flags,const void *lmm,int lmmlen) {struct mdt_rec_create *rec;
@@ -234,6 +234,7 @@ void mdc_open_pack(struct ptlrpc_request *req, struct md_op_data *op_data, rec->cr_suppgid2 = op_data->op_suppgids[1 ]; rec->cr_bias = op_data->op_bias; rec->cr_umask =current_umask (); + rec->cr_old_handle = op_data->op_handle;mdc_pack_capa (req, &RMF_CAPA1 , op_data->op_capa1);/* the next buffer is child capa, which is used for replay,
drivers/staging/lustre/lustre/mdc/mdc_locks.c +26 -13
@@ -75,6 +75,12 @@ EXPORT_SYMBOL(it_clear_disposition);int it_open_error (int phase,struct lookup_intent *it) { +if (it_disposition (it,DISP_OPEN_LEASE )) { +if (phase >=DISP_OPEN_LEASE ) +return it->d.lustre.it_status; +else +return 0 ; + }if (it_disposition (it,DISP_OPEN_OPEN )) {if (phase >=DISP_OPEN_OPEN )return it->d.lustre.it_status;
@@ -281,14 +287,21 @@ static struct ptlrpc_request *mdc_intent_open_pack(struct obd_export *exp,/* XXX: openlock is not cancelled for cross-refs. */ /* If inode is known, cancel conflicting OPEN locks. */ if (fid_is_sane (&op_data->op_fid2)) { -if (it->it_flags & (FMODE_WRITE |MDS_OPEN_TRUNC )) - mode =LCK_CW ; +if (it->it_flags &MDS_OPEN_LEASE ) {/* try to get lease */ +if (it->it_flags &FMODE_WRITE ) + mode =LCK_EX ; +else + mode =LCK_PR ; + }else { +if (it->it_flags & (FMODE_WRITE |MDS_OPEN_TRUNC )) + mode =LCK_CW ;#ifdef FMODE_EXEC - else if (it->it_flags & FMODE_EXEC) - mode =LCK_PR ; + else if (it->it_flags & FMODE_EXEC) + mode =LCK_PR ;#endif - else - mode =LCK_CR ; + else + mode =LCK_CR ; + } count =mdc_resource_get_unused (exp, &op_data->op_fid2, &cancels, mode,MDS_INODELOCK_OPEN );
@@ -1065,10 +1078,10 @@ int mdc_intent_lock(struct obd_export *exp, struct md_op_data *op_data,LASSERT (it);CDEBUG (D_DLMTRACE ,"(name: %.*s," DFID ") in obj " DFID -", intent: %s flags %#o\n" , op_data->op_namelen, - op_data->op_name,PFID (&op_data->op_fid2), -PFID (&op_data->op_fid1),ldlm_it2str (it->it_op), - it->it_flags); +", intent: %s flags %#Lo\n" , op_data->op_namelen, + op_data->op_name,PFID (&op_data->op_fid2), +PFID (&op_data->op_fid1),ldlm_it2str (it->it_op), + it->it_flags); lockh.cookie =0 ; if (fid_is_sane(&op_data->op_fid2) &&
@@ -1194,9 +1207,9 @@ int mdc_intent_getattr_async(struct obd_export *exp,int rc =0 ;__u64 flags =LDLM_FL_HAS_INTENT ; -CDEBUG (D_DLMTRACE ,"name: %.*s in inode " DFID ", intent: %s flags %#o\n" , - op_data->op_namelen, op_data->op_name,PFID (&op_data->op_fid1), -ldlm_it2str (it->it_op), it->it_flags); +CDEBUG (D_DLMTRACE ,"name: %.*s in inode " DFID ", intent: %s flags %#Lo\n" , + op_data->op_namelen, op_data->op_name,PFID (&op_data->op_fid1), +ldlm_it2str (it->it_op), it->it_flags);fid_build_reg_res_name (&op_data->op_fid1, &res_id); req =mdc_intent_getattr_pack (exp, it, op_data);
Signature · 1 line
1.7.9.5
Re: [PATCH 12/40] staging/lustre/nfs: writing to new files will return ENOENT
In reply to Cheng ShaoQuoted 4 lines · Cheng Shao
Hi Peng,
This patch was eventually reverted as it was causing interoperability issues with 2.1 Lustre server. We are working on the new patch, so please don't include this one.
Thanks for the notification. I didn't see the reverting patch in next 200 patches in Lustre tree so it must have happened recently.
Greg, please drop this one. I verified that the rest of the patchset can be applied cleanly.
Thanks,
Tao
Quoted 64 lines
Thanks,
- ChengQuoted 61 lines · Peng Tao
From: Patrick Farrell <[email protected]>
This happend with SLES11SP2 Lustre client, which in turn acts as an NFS server, exporting a subtree of an Lustre fs through NFS.
We detected that whenever we are writing to a new file using, fx, 'echo blah > newfile', it will return ENOENT error. We found out that this was caused by the anonymous dentry. In SLESS11SP2, anonymous dentries are assigned '/' as the name, instead of an empty string. When MDT handles the
intent_opencall, it will look up the obj by the name if it is not an empty string, and thus couldn't find it.As
MDS_OPEN_BY_FIDis always set on this request, we never need to send the name in this request. The fid is already available and should be used in case the file has been renamed.Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-3544
Lustre-change: http://review.whamcloud.com/6920Signed-off-by Cheng Shao <[email protected]>Signed-off-by Patrick Farrell <[email protected]>Reviewed-by Bob Glossman <[email protected]>Reviewed-by Alexey Shvetsov <[email protected]>Reviewed-by Lai Siyao <[email protected]>Reviewed-by James Simmons <[email protected]>Reviewed-by Oleg Drokin <[email protected]>Signed-off-by Peng Tao <[email protected]>Signed-off-by Andreas Dilger <[email protected]>---
drivers/staging/lustre/lustre/llite/file.c| 5 ++---1 file changed, 2 insertions(+), 3 deletions(-)drivers/staging/lustre/lustre/llite/file.c +2
-4@@ -368,8 +368,6 @@ static int ll_intent_file_open(struct file *file, void *lmm, {struct ll_sb_info *sbi =ll_i2sbi (file->f_dentry->d_inode);struct dentry *parent = file->f_dentry->d_parent; -const char *name = file->f_dentry->d_name.name; -const int len = file->f_dentry->d_name.len;struct md_op_data *op_data;struct ptlrpc_request *req;__u32 opc =LUSTRE_OPC_ANY ;@@ -394,8 +392,9 @@ static int ll_intent_file_open(struct file *file, void *lmm, } op_data =ll_prep_md_op_data (NULL , parent->d_inode, - file->f_dentry->d_inode, name, len, + file->f_dentry->d_inode,NULL ,0 ,O_RDWR , opc,NULL ); +if (IS_ERR (op_data))return PTR_ERR (op_data); --1.7.9.5
Greg Kroah-Hartman
maintainerQuoted 186 lines · Peng Tao
[sadly git-send-email died sending the patchset... so resending]
Hi Greg,
Here are 40 patches. Three cleanup patches and 37 patches ported from Lustre master to commit
LU-2940 llite: Fix for oops invvp_pgcache_show()There are two more patchsets that I will send out following up.
Thanks,
TaoCc Andreas Dilger <[email protected]>Amir Shehata (2):
staging/lustre/lnet: Fix assert on empty group in selftest module staging/lustre/ptlrpc: Fix a crash when dereferencing NULL pointerAndreas Dilger (2):
staging/lustre/ldlm: fix resource/fid check, use DLDLMRES staging/lustre/idl: remove LASSERT/CLASSERT fromlustre_idl.hAndrew Perepechko (1):
staging/lustre/llite: extended attribute cacheAndriy Skulysh (2):
staging/lustre/ptlrpc: Fix race duringexp_flock_hashcreation staging/lustre/ptlrpc: flock deadlock detection does not workArtem Blagodarenko (1):
staging/lustre/mgs:set_param-P option that sets value permanentlyDmitry Eremin (2):
staging/lustre/build: clean up unused variables and dead code staging/lustre/build: fix compilation issue withis_compat_taskDoug Oucharek (1):
staging/lustre/lnet: Add LNet Router Priority parameterFan Yong (2):
staging/lustre/scrub: OI scrub on OST staging/lustre/scrub: control OI scrub on OST from user spaceJC Lafoucriere (5):
staging/lustre/llite: Access to released file trigs a restore staging/lustre/mdt: HSM coordinator client interface staging/lustre/mdt: HSM coordinator agent interface staging/lustre/api: HSM import uses new released pattern staging/lustre/utils: HSM Posix CopyToolJames Simmons (2):
staging/lustre/autoconf: remove vectored fops tests staging/lustre/autoconf: removeLIBCFS_HAVE_IS_COMPAT_TASKtestJinshan Xiong (2):
staging/lustre/hsm: Implementation of exclusive open staging/lustre/hsm: Addhsm_releasefeature.John L. Hammond (7):
staging/lustre: validate open handle cookies staging/lustre/llite: use correct FID inll_och_fill()staging/lustre/lov: convert magic to host-endian inlov_dump_lmm()staging/lustre/mdc: prevent fall through inmdc_iocontrol()staging/lustre/lu: shrinklu_objectby 8 bytes onx86_64staging/lustre/llite: don't check forO_CREATinit_create_modestaging/lustre/llite: pass correct pointer toobd_iocontrol()Lai Siyao (1):
staging/lustre/llite: removell_d_root_opsLi Xi (1):
staging/lustre/llog: fix return value ofllog_alloc_handleMikhail Pershin (4):
staging/lustre/server: use unified request handler for MGS staging/lustre/llog: MGC to use OSD API for backup logs staging/lustre/target: move OUT to the unified target code staging/lustre/seq: unified SEQ handlerNathaniel Clark (1):
staging/lustre/xattr: separate ACL and XATTR cachesPatrick Farrell (1):
staging/lustre/nfs: writing to new files will return ENOENTPeng Tao (3):
staging/lustre/llite: restorell_fiemap
staging/lustre: removelu_target.h
staging/lustre: removellog_server.c.../staging/lustre/include/linux/libcfs/curproc.h | 1 - .../staging/lustre/include/linux/libcfs/libcfs.h | 2 - .../lustre/include/linux/libcfs/libcfs_ioctl.h | 1 + .../staging/lustre/include/linux/lnet/lib-lnet.h | 5 +- .../staging/lustre/include/linux/lnet/lib-types.h | 2 +- drivers/staging/lustre/lnet/lnet/api-ni.c | 5 +- drivers/staging/lustre/lnet/lnet/config.c | 39 +- drivers/staging/lustre/lnet/lnet/lib-move.c | 6 + drivers/staging/lustre/lnet/lnet/router.c | 19 +- drivers/staging/lustre/lnet/lnet/router_proc.c | 16 +- drivers/staging/lustre/lnet/selftest/conctl.c | 56 +- drivers/staging/lustre/lnet/selftest/conrpc.c | 2 +- drivers/staging/lustre/lnet/selftest/console.c | 105 ++-- drivers/staging/lustre/lnet/selftest/console.h | 6 +- drivers/staging/lustre/lnet/selftest/rpc.c | 2 - drivers/staging/lustre/lnet/selftest/selftest.h | 3 - drivers/staging/lustre/lnet/selftest/timer.c | 6 +- drivers/staging/lustre/lustre/include/cl_object.h | 6 +- drivers/staging/lustre/lustre/include/dt_object.h | 2 +- .../lustre/lustre/include/linux/lustre_compat25.h | 4 +- .../lustre/lustre/include/linux/lustre_intent.h | 2 +- .../lustre/lustre/include/linux/lustre_lite.h | 1 + drivers/staging/lustre/lustre/include/lu_object.h | 19 - drivers/staging/lustre/lustre/include/lu_target.h | 91 --- .../lustre/lustre/include/lustre/lustre_idl.h | 111 ++-- .../lustre/lustre/include/lustre/lustre_user.h | 63 +- .../lustre/lustre/include/lustre/lustreapi.h | 208 ++++--- drivers/staging/lustre/lustre/include/lustre_cfg.h | 2 + .../staging/lustre/lustre/include/lustre_disk.h | 2 + .../lustre/lustre/include/lustre_dlm_flags.h | 24 +- drivers/staging/lustre/lustre/include/lustre_fid.h | 6 - drivers/staging/lustre/lustre/include/lustre_ha.h | 3 - .../staging/lustre/lustre/include/lustre_handles.h | 5 +- drivers/staging/lustre/lustre/include/lustre_lib.h | 11 +- drivers/staging/lustre/lustre/include/lustre_log.h | 13 +- drivers/staging/lustre/lustre/include/lustre_net.h | 19 +- .../lustre/lustre/include/lustre_req_layout.h | 7 + drivers/staging/lustre/lustre/include/md_object.h | 4 +- drivers/staging/lustre/lustre/include/obd.h | 15 +- drivers/staging/lustre/lustre/include/obd_class.h | 9 +- .../staging/lustre/lustre/include/obd_support.h | 10 + drivers/staging/lustre/lustre/lclient/lcommon_cl.c | 6 + .../staging/lustre/lustre/lclient/lcommon_misc.c | 4 +- drivers/staging/lustre/lustre/ldlm/ldlm_flock.c | 69 ++- drivers/staging/lustre/lustre/ldlm/ldlm_lock.c | 7 +- drivers/staging/lustre/lustre/ldlm/ldlm_lockd.c | 47 +- drivers/staging/lustre/lustre/ldlm/ldlm_request.c | 39 +- drivers/staging/lustre/lustre/ldlm/ldlm_resource.c | 19 +- .../lustre/lustre/libcfs/linux/linux-curproc.c | 13 - drivers/staging/lustre/lustre/libcfs/workitem.c | 55 +- drivers/staging/lustre/lustre/llite/Makefile | 2 +- drivers/staging/lustre/lustre/llite/dcache.c | 77 +-- drivers/staging/lustre/lustre/llite/dir.c | 47 +- drivers/staging/lustre/lustre/llite/file.c | 622 ++++++++++++++++++-- .../staging/lustre/lustre/llite/llite_internal.h | 59 +- drivers/staging/lustre/lustre/llite/llite_lib.c | 91 ++- drivers/staging/lustre/lustre/llite/llite_nfs.c | 6 +- drivers/staging/lustre/lustre/llite/lproc_llite.c | 36 ++ drivers/staging/lustre/lustre/llite/namei.c | 53 +- drivers/staging/lustre/lustre/llite/statahead.c | 7 +- drivers/staging/lustre/lustre/llite/super25.c | 4 + drivers/staging/lustre/lustre/llite/vvp_io.c | 54 +- drivers/staging/lustre/lustre/llite/vvp_object.c | 2 +- drivers/staging/lustre/lustre/llite/xattr.c | 105 ++-- drivers/staging/lustre/lustre/llite/xattr_cache.c | 559 ++++++++++++++++++ .../staging/lustre/lustre/lov/lov_cl_internal.h | 16 + drivers/staging/lustre/lustre/lov/lov_internal.h | 4 +- drivers/staging/lustre/lustre/lov/lov_io.c | 15 +- drivers/staging/lustre/lustre/lov/lov_object.c | 35 +- drivers/staging/lustre/lustre/lov/lov_pack.c | 20 +- drivers/staging/lustre/lustre/mdc/mdc_internal.h | 5 +- drivers/staging/lustre/lustre/mdc/mdc_lib.c | 31 +- drivers/staging/lustre/lustre/mdc/mdc_locks.c | 104 +++- drivers/staging/lustre/lustre/mdc/mdc_reint.c | 2 +- drivers/staging/lustre/lustre/mdc/mdc_request.c | 116 ++-- drivers/staging/lustre/lustre/mgc/libmgc.c | 3 - drivers/staging/lustre/lustre/mgc/mgc_request.c | 492 ++++++++++------ drivers/staging/lustre/lustre/obdclass/genops.c | 2 +- drivers/staging/lustre/lustre/obdclass/llog.c | 214 ++++--- .../staging/lustre/lustre/obdclass/local_storage.c | 9 +- .../staging/lustre/lustre/obdclass/local_storage.h | 3 + .../lustre/lustre/obdclass/lprocfs_status.c | 5 +- drivers/staging/lustre/lustre/obdclass/lu_object.c | 26 +- .../lustre/lustre/obdclass/lustre_handles.c | 4 +- drivers/staging/lustre/lustre/obdclass/md_attrs.c | 4 +- .../staging/lustre/lustre/obdclass/obd_config.c | 50 +- drivers/staging/lustre/lustre/obdclass/obd_mount.c | 3 + .../staging/lustre/lustre/obdecho/echo_client.c | 2 +- drivers/staging/lustre/lustre/ptlrpc/Makefile | 2 +- drivers/staging/lustre/lustre/ptlrpc/client.c | 10 +- drivers/staging/lustre/lustre/ptlrpc/import.c | 2 - drivers/staging/lustre/lustre/ptlrpc/layout.c | 63 +- drivers/staging/lustre/lustre/ptlrpc/llog_server.c | 450 -------------- .../staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c | 4 +- drivers/staging/lustre/lustre/ptlrpc/niobuf.c | 16 +-
Quoted 8 lines
.../staging/lustre/lustre/ptlrpc/pack_generic.c | 59 +- drivers/staging/lustre/lustre/ptlrpc/pinger.c | 65 -- drivers/staging/lustre/lustre/ptlrpc/ptlrpcd.c | 44 +- drivers/staging/lustre/lustre/ptlrpc/wiretest.c | 60 +- 99 files changed, 3043 insertions(+), 1793 deletions(-) delete mode 100644 drivers/staging/lustre/lustre/include/lu_target.h create mode 100644 drivers/staging/lustre/lustre/llite/xattr_cache.c delete mode 100644 drivers/staging/lustre/lustre/ptlrpc/llog_server.c
Some of the stuff here looks a lot like "new features / functionality" to me, insead of just cleanups and fixes.
Ideally, you all would only work on cleanups and getting this code out of staging, and not add new things while it's sitting here. But I know that's going to be hard as you all are still working with an external tree.
That external tree is going to be tough to keep up over time, I _strongly_ suggest you drop it as soon as possible and only use the upstream tree. Otherwise this whole thing really isn't going to work out at all.
I'll go dig through these and see what they look like, thanks...
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH 12/40] staging/lustre/nfs: writing to new files will return ENOENT
In reply to Peng TaoQuoted 10 lines · Peng Tao
Quoted 4 lines · Cheng Shao
Hi Peng,
This patch was eventually reverted as it was causing interoperability issues with 2.1 Lustre server. We are working on the new patch, so please don't include this one.
Thanks for the notification. I didn't see the reverting patch in next 200 patches in Lustre tree so it must have happened recently.
Greg, please drop this one. I verified that the rest of the patchset can be applied cleanly.
Now dropped, thanks.
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
In reply to Peng TaoQuoted 10 lines · Peng Tao
From: JC Lafoucriere <[email protected]>
When a client accesses data in a released file,
or truncate it, client must trig a restore request. During this restore, the client must not glimpse and must use size from MDT. To bring the "restore is running" information on the client we add a newt_statebit field tomdt_infowhich will be used to carry transient file state. To memorise this information in the inode we add a new flagLLIF_FILE_RESTORING.
This patch also does other things not mentioned here (coding style cleanups), which isn't allowed in a single patch (only do one thing per patch, and never not document what you are doing...)
It also adds checkpatch warnings, which I will not accept in patches at all here. People are spending a lot of time cleaning up the coding style issues, please NEVER add new ones, that just causes more work to be needed to be done, and for people to have to go back and reclean files they have already cleaned up.
So, sorry, I have to stop here at this series. I've applied the first 3 to the opw-next branch of staging.git so they can live somewhere until 3.13-rc1 is out.
I know you spent a lot of time making these 120 patches to send me, but that too is crazy. You shouldn't wait that long to get feedback and send patches to me at all. Please send them in smaller series, with less time between patch submissions.
So, care to just send me 10 patches or so now, which I can review and accept if good, and we can sync up and continue from there?
thanks,
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH 07/40] staging/lustre/hsm: Implementation of exclusive open
In reply to Peng TaoQuoted 4 lines · Peng Tao
From: Jinshan Xiong <[email protected]>
In this patch, a framework of lease is implemented. However, only exclusive lease is supported right now.
Which is a problem, why?
Why do you want to add this new functionality? What is it good for? Who will use it?
Oh, and the patch doesn't apply to my tree, AND it contains a bunch of foolish coding style errors, so even if I did understand what this was for, and why we wanted it, I couldn't.
Please be more careful, I'm really stopping here now on reviewing these lustre patches. Please clean them up, resync, and try again, I've now purged them all from my mboxes, sorry.
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH 05/40] staging/lustre: validate open handle cookies
In reply to Peng TaoQuoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.
Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.
This changelog entry doesn't even match up with the code below. ALl callers of class_handle2object are passing NULL here, which makes this patch pretty pointless, right?
And that's a _very_ generic global symbol name, please don't do that, it needs to be "lustre_*" at the front to even expect it to be acceptable.
thanks,
greg k-h
Quoted 8 lines · Greg Kroah-Hartman
Some of the stuff here looks a lot like "new features / functionality" to me, insead of just cleanups and fixes.
Ideally, you all would only work on cleanups and getting this code out of staging, and not add new things while it's sitting here. But I know that's going to be hard as you all are still working with an external tree.
Quoted 5 lines
That external tree is going to be tough to keep up over time, I _strongly_ suggest you drop it as soon as possible and only use the upstream tree. Otherwise this whole thing really isn't going to work out at all.
There are many users of the external tree so we cannot just abandon it, especially that the upstream client is not shipped in any distribution yet. Thanks for your understanding.
Quoted 2 lines
I'll go dig through these and see what they look like, thanks...
Thanks,
Tao
Re: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
In reply to Greg Kroah-HartmanQuoted 12 lines · Greg Kroah-Hartman
Quoted 10 lines · Peng Tao
From: JC Lafoucriere <[email protected]>
When a client accesses data in a released file,
or truncate it, client must trig a restore request. During this restore, the client must not glimpse and must use size from MDT. To bring the "restore is running" information on the client we add a newt_statebit field tomdt_infowhich will be used to carry transient file state. To memorise this information in the inode we add a new flagLLIF_FILE_RESTORING.
Quoted 4 lines
This patch also does other things not mentioned here (coding style cleanups), which isn't allowed in a single patch (only do one thing per patch, and never not document what you are doing...)
I ported the original patch with as least modification as possible. Guess I need to split this one (and the alike) so that it follows upstream patch rules.
Quoted 6 lines
It also adds checkpatch warnings, which I will not accept in patches at all here. People are spending a lot of time cleaning up the coding style issues, please NEVER add new ones, that just causes more work to be needed to be done, and for people to have to go back and reclean files they have already cleaned up.
sorry... I will be more careful about this.
Quoted 4 lines
So, sorry, I have to stop here at this series. I've applied the first 3 to the opw-next branch of staging.git so they can live somewhere until 3.13-rc1 is out.
Thanks!
Quoted 8 lines
I know you spent a lot of time making these 120 patches to send me, but that too is crazy. You shouldn't wait that long to get feedback and send patches to me at all. Please send them in smaller series, with less time between patch submissions.
So, care to just send me 10 patches or so now, which I can review and accept if good, and we can sync up and continue from there?
OK. I will split the patchsets into smaller ones. Thanks for reviewing!
Thanks,
Tao
Re: [PATCH 05/40] staging/lustre: validate open handle cookies
In reply to Greg Kroah-HartmanQuoted 7 lines · Greg Kroah-Hartman
Quoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.
Quoted 7 lines
Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
Here the void * is not used to pass any structures. It just intends to save any pointer that is viewed as owner of the handle. Does it make sense?
Maybe we can just make it unsigned long and type cast the owner pointer to unsigned long as well. But I guess that is even more hacky...
Quoted 10 lines
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.
Quoted 4 lines
This changelog entry doesn't even match up with the code below. ALl callers of
class_handle2objectare passing NULL here, which makes this patch pretty pointless, right?
The extra pointer argument is in fact used by server code. The function is shared by client and server and we want to keep it sync in the two places.
Quoted 3 lines
And that's a _very_ generic global symbol name, please don't do that, it needs to be "lustre_*" at the front to even expect it to be acceptable.
OK. I'll rename it.
Thanks,
Tao
Re: [PATCH 07/40] staging/lustre/hsm: Implementation of exclusive open
In reply to Greg Kroah-HartmanQuoted 11 lines · Greg Kroah-Hartman
Quoted 4 lines · Peng Tao
From: Jinshan Xiong <[email protected]>
In this patch, a framework of lease is implemented. However, only exclusive lease is supported right now.
Which is a problem, why?
Why do you want to add this new functionality? What is it good for? Who will use it?
Only brief info for the patch in included in the commit message. The long story is saved in the external link
https://jira.hpdd.intel.com/browse/LU-2919. I guess I need to pull more bits to the commit message to make it more understandable.
Quoted 4 lines
Oh, and the patch doesn't apply to my tree, AND it contains a bunch of foolish coding style errors, so even if I did understand what this was for, and why we wanted it, I couldn't.
I'll fix up. sorry.
Thanks,
Tao
Greg Kroah-Hartman
maintainerQuoted 10 lines · Peng Tao
Quoted 8 lines · Greg Kroah-Hartman
Some of the stuff here looks a lot like "new features / functionality" to me, insead of just cleanups and fixes.
Ideally, you all would only work on cleanups and getting this code out of staging, and not add new things while it's sitting here. But I know that's going to be hard as you all are still working with an external tree.
Quoted 8 lines
Quoted 5 lines
That external tree is going to be tough to keep up over time, I _strongly_ suggest you drop it as soon as possible and only use the upstream tree. Otherwise this whole thing really isn't going to work out at all.
There are many users of the external tree so we cannot just abandon it, especially that the upstream client is not shipped in any distribution yet. Thanks for your understanding.
What is keeping people using that tree? Support for older/distro kernels?
Is it the fact that the server code isn't in the kernel? Should that be added now too so that we can get a proper view of what can and can not be changed? Some of your patches are showing that things are shared by the two chunks of code, so does that mean if I delete things in the client code that don't look to be used by anything, you will have problems because the server now breaks?
I think it's time to just merge the server and deal with the whole thing all at once, otherwise this dependancy on your external tree, and external code to the kernel, is going to doom the ability to ever get this code cleaned up and merged properly.
thanks,
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH 05/40] staging/lustre: validate open handle cookies
In reply to Peng TaoQuoted 11 lines · Peng Tao
Quoted 9 lines · Greg Kroah-Hartman
Quoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.Ick ick ick.
Quoted 8 lines
Quoted 5 lines
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
Here the void * is not used to pass any structures. It just intends to save any pointer that is viewed as owner of the handle. Does it make sense?
Yes, but you shouldn't use a void * for it. Use the _real_ data structure it is pointing to.
And "handles" suck, please don't use them, it's not a Linux model at all, for good reasons (i.e. we have the ability to know what is being passed around.)
Quoted 3 lines
Maybe we can just make it unsigned long and type cast the owner pointer to unsigned long as well. But I guess that is even more hacky...
Yes, that's not the correct solution either :)
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.
Quoted 7 lines
Quoted 4 lines
This changelog entry doesn't even match up with the code below. ALl callers of
class_handle2objectare passing NULL here, which makes this patch pretty pointless, right?The extra pointer argument is in fact used by server code. The function is shared by client and server and we want to keep it sync in the two places.
Ok, this is proof we need the server code in the kernel, because as it is, I can't take this patch because it doesn't do anything at all, and the changelog lies.
Dealing with an external dependancy on kernel code does not work at all, sorry. Let's just merge both into staging and deal with the cleanup properly. Otherwise this is going to be impossible to do, and we should just delete the client code as I sure can't work on it as is now.
sorry,
greg k-h
Re: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
In reply to Greg Kroah-HartmanQuoted 16 lines · Greg Kroah-Hartman
Quoted 10 lines · Peng Tao
From: JC Lafoucriere <[email protected]>
When a client accesses data in a released file,
or truncate it, client must trig a restore request. During this restore, the client must not glimpse and must use size from MDT. To bring the "restore is running" information on the client we add a newt_statebit field tomdt_infowhich will be used to carry transient file state. To memorise this information in the inode we add a new flagLLIF_FILE_RESTORING.This patch also does other things not mentioned here (coding style cleanups), which isn't allowed in a single patch (only do one thing per patch, and never not document what you are doing...)
Quoted 5 lines
It also adds checkpatch warnings, which I will not accept in patches at all here. People are spending a lot of time cleaning up the coding style issues, please NEVER add new ones, that just causes more work to be needed to be done, and for people to have to go back and reclean files they have already cleaned up.
I'm fine to clean up the coding style issues in these patches.
Quoted 8 lines
So, sorry, I have to stop here at this series. I've applied the first 3 to the opw-next branch of staging.git so they can live somewhere until 3.13-rc1 is out.
I know you spent a lot of time making these 120 patches to send me, but that too is crazy. You shouldn't wait that long to get feedback and send patches to me at all. Please send them in smaller series, with less time between patch submissions.
The reason that there are so many patches in a burst is that we are also developing new features and fixing bugs in parallel with the kernel, but they also need to be tested a lot before they are released. It's not any different from kernel patches testing on -next or -mm for a few months before they are pushed to the mainline kernel in a batch.
The out-of-tree development can't really stop for a year while the kernel client code is cleaned up. If the feature patches don't land into the kernel client for a year (or however long it takes to do all the cleanup), then it will become outdated and the whole reason for adding the client into the kernel is lost.
Quoted 7 lines
Quoted 3 lines
There are many users of the external tree so we cannot just abandon it, especially that the upstream client is not shipped in any distribution yet. Thanks for your understanding.
What is keeping people using that tree? Support for older/distro kernels?
Support for distro kernels is a big part of it. Most HPC sites use vendor kernels of various vintages, and we need to keep the code working for those sites. We also need to continue developing features needed by ever-larger clusters, fixing bugs, etc. Eventually, when Lustre is in the kernel proper,
it will also be available in future distro kernels and we can eventually stop supporting the out-of-tree code. I expect that to be 3+ years away.
Quoted 1 line
Is it the fact that the server code isn't in the kernel?
Not really. Lustre servers are on separate servers with vendor kernels (ancient by your standards), and there isn't really any demand for using newer kernels on those nodes. Most importantly, the out-of-tree branch is where all of the new feature development is being done. That also means if feature patches don't land into the kernel it just makes the kernel version less attractive for users.
Quoted 5 lines
Should that be added now too so that we can get a proper view of what
can and can not be changed? Some of your patches are showing that things are shared by the two chunks of code, so does that mean if I delete things in the client code that don't look to be used by anything, you will have problems because the server now breaks?
Adding the server code to staging would mean another 150kLOC in staging. We also haven't cleaned that code up even nearly as much as the client code, so it isn't really even ready to go into staging either. I don't think you or anyone else would be happy to see that code yet, so I don't think there is a real advantage to do so.
Deleting unused code in the kernel client isn't fatal, since we'll still have the out-of-tree version, but we're trying to keep the code in sync as much as possible so that it is easier to port patches in both directions. If code is deleted it means more that needs to be added back later when the server eventually does get merged, and more effort to resolve patch conflicts.
Quoted 4 lines
I think it's time to just merge the server and deal with the whole thing all at once, otherwise this dependancy on your external tree, and external code to the kernel, is going to doom the ability to ever get this code cleaned up and merged properly.
Regardless of whether the Lustre server is added to the kernel or not, we need to maintain support for the older kernels, so there will need to be an external tree for years to come until Lustre is available in vendor kernels. I'm sure we can't merge in the kernel-version-portable code into the kernel, so there isn't really any choice but to try and keep both trees in sync.
Cheers, Andreas
Signature · 4 lines
Andreas Dilger Lustre Software Architect Intel High Performance Data Division
Re: [PATCH 05/40] staging/lustre: validate open handle cookies
In reply to Greg Kroah-HartmanQuoted 7 lines · Greg Kroah-Hartman
Quoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.
Quoted 6 lines
Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
The portals_handle is used as a generic type for objects referenced over the network, like a file handle. The "owner" parameter is just used as an extra check that the cookie passed from the client is actually a valid value for the code in which it is being used (e.g. metadata or data object). It isn't actually dereferenced by anything, it is just like a magic value.
Quoted 10 lines
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.
Quoted 3 lines
This changelog entry doesn't even match up with the code below. ALl callers of
class_handle2objectare passing NULL here, which makes this patch pretty pointless, right?
As Tao wrote, this is the patch summary that matches what was committed in our own tree and in this case mostly describes the changes made on the server. Keeping the same commits and comments in both trees makes it easier to keep the code in sync.
Quoted 2 lines
And that's a _very_ generic global symbol name, please don't do that, it needs to be "lustre_*" at the front to even expect it to be acceptable.
There's already something else called a "lustre_handle", so what about obdclass_handle2object(), since this is in the obdclass module?
Cheers, Andreas
Signature · 4 lines
Andreas Dilger Lustre Software Architect Intel High Performance Data Division
Greg Kroah-Hartman
maintainerRe: [PATCH 05/40] staging/lustre: validate open handle cookies
Quoted 17 lines · Andreas
Quoted 7 lines · Greg Kroah-Hartman
Quoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.Quoted 6 lines
Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
Quoted 6 lines
The
portals_handleis used as a generic type for objects referenced over the network, like a file handle. The "owner" parameter is just used as an extra check that the cookie passed from the client is actually a valid value for the code in which it is being used (e.g. metadata or data object). It isn't actually dereferenced by anything, it is just like a magic value.
Then make it an explicit type, not a void *.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.
Quoted 8 lines
Quoted 3 lines
This changelog entry doesn't even match up with the code below. ALl callers of
class_handle2objectare passing NULL here, which makes this patch pretty pointless, right?As Tao wrote, this is the patch summary that matches what was committed in our own tree and in this case mostly describes the changes made on the server. Keeping the same commits and comments in both trees makes it easier to keep the code in sync.
Ok, but as it is, this patch does nothing to the client code, so how can I accept it? A function that is only ever called with NULL as an option is ripe for cleanup in my eyes.
Quoted 5 lines
Quoted 2 lines
And that's a _very_ generic global symbol name, please don't do that, it needs to be "lustre_*" at the front to even expect it to be acceptable.
There's already something else called a "
lustre_handle", so what aboutobdclass_handle2object(), since this is in the obdclass module?
I don't care, but make it unique to the lustre subsystem somehow. Haveing it start with "class_" looks like it belongs to the driver core which isn't ok.
thanks,
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
In reply to Dilger, AndreasQuoted 9 lines · Andreas
Quoted 8 lines
So, sorry, I have to stop here at this series. I've applied the first 3 to the opw-next branch of staging.git so they can live somewhere until 3.13-rc1 is out.
I know you spent a lot of time making these 120 patches to send me, but that too is crazy. You shouldn't wait that long to get feedback and send patches to me at all. Please send them in smaller series, with less time between patch submissions.
Quoted 5 lines
The reason that there are so many patches in a burst is that we are also developing new features and fixing bugs in parallel with the kernel, but they also need to be tested a lot before they are released. It's not any different from kernel patches testing on -next or -mm for a few months before they are pushed to the mainline kernel in a batch.
It's very different in that you are expecting me to suddenly accept this huge burst of patches all at once, and I can't review them properly. As this patch series shows, there was a problem in the 4rd patch in a 100+ series, which means that I couldn't take them all. So you wasted a bunch of work preparing those other 96 patches.
Send them to me in short chunks, that way you don't waste time, and you don't take so long between patch acceptance.
Quoted 5 lines
The out-of-tree development can't really stop for a year while the kernel client code is cleaned up. If the feature patches don't land into the kernel client for a year (or however long it takes to do all the cleanup), then it will become outdated and the whole reason for adding the client into the kernel is lost.
But you understand my hesitation at taking any new features when there really has not been any attempt by your team to do much cleanup and fixes of the code at all, right? It feels like I have done more cleanup personally than anyone on your teams, is this not true?
So, why would I believe that you all are really going to start doing the major cleanup work on this code that is so obviously needed? Why would I take new features that you are spending your time on instead?
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
There are many users of the external tree so we cannot just abandon it, especially that the upstream client is not shipped in any distribution yet. Thanks for your understanding.
What is keeping people using that tree? Support for older/distro kernels?
Quoted 7 lines
Support for distro kernels is a big part of it. Most HPC sites use vendor kernels of various vintages, and we need to keep the code working for those sites. We also need to continue developing features needed by ever-larger clusters, fixing bugs, etc. Eventually, when Lustre is in the kernel proper,
it will also be available in future distro kernels and we can eventually stop supporting the out-of-tree code. I expect that to be 3+ years away.
So, originally you said it would take about a year to get this out of staging. It's been 6 months already with no real progress made with the exception of having interns do cleanup and me doing some basic wrapper function removals. What's the plan for the next 6 months? Based on these 100+ patches, I don't see any major changes that are happening to get this cleaned up properly (or did I miss those patches in this huge patchbomb?)
Quoted 21 lines
Quoted 1 line
Is it the fact that the server code isn't in the kernel?
Not really. Lustre servers are on separate servers with vendor kernels (ancient by your standards), and there isn't really any demand for using newer kernels on those nodes. Most importantly, the out-of-tree branch is where all of the new feature development is being done. That also means if feature patches don't land into the kernel it just makes the kernel version less attractive for users.
Quoted 5 lines
Should that be added now too so that we can get a proper view of what
can and can not be changed? Some of your patches are showing that things are shared by the two chunks of code, so does that mean if I delete things in the client code that don't look to be used by anything, you will have problems because the server now breaks?Adding the server code to staging would mean another 150kLOC in staging. We also haven't cleaned that code up even nearly as much as the client code, so it isn't really even ready to go into staging either. I don't think you or anyone else would be happy to see that code yet, so I don't think there is a real advantage to do so.
Quoted 6 lines
Deleting unused code in the kernel client isn't fatal, since we'll still have the out-of-tree version, but we're trying to keep the code in sync as much as possible so that it is easier to port patches in both directions. If code is deleted it means more that needs to be added back later when the server eventually does get merged, and more effort to resolve patch conflicts.
But look at the function that caused this original question to be asked. You want an api change that I would never accept as it doesn't do anything to the client code at all. How do I "know" this is acceptable and needed if I can't see the server side? How would I "know" not to take a cleanup patch that reverted this change if I can't see the server side at the same time?
150k of code is not a big deal to me, I can easily take it. If it means that we have a real chance at getting this all fixed up properly, I say to do it, otherwise I really don't think this project will ever succeed, sorry.
greg k-h
Re: [PATCH 05/40] staging/lustre: validate open handle cookies
In reply to Greg Kroah-HartmanQuoted 41 lines · Greg Kroah-Hartman
Quoted 17 lines · Andreas
Quoted 7 lines · Greg Kroah-Hartman
Quoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.Quoted 6 lines
Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
Quoted 6 lines
The
portals_handleis used as a generic type for objects referenced over the network, like a file handle. The "owner" parameter is just used as an extra check that the cookie passed from the client is actually a valid value for the code in which it is being used (e.g. metadata or data object). It isn't actually dereferenced by anything, it is just like a magic value.Then make it an explicit type, not a void *.
Quoted 14 lines
Quoted 13 lines
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.This changelog entry doesn't even match up with the code below. ALl callers of
class_handle2objectare passing NULL here, which makes this patch pretty pointless, right?
Quoted 9 lines
Quoted 4 lines
As Tao wrote, this is the patch summary that matches what was committed in our own tree and in this case mostly describes the changes made on the server. Keeping the same commits and comments in both trees makes it easier to keep the code in sync.
Ok, but as it is, this patch does nothing to the client code, so how can I accept it? A function that is only ever called with NULL as an option is ripe for cleanup in my eyes.
How about adding a comment above the function to note that this extra argument is used by server code and please don't remove it?
Thanks,
Tao
Re: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
In reply to Greg Kroah-HartmanQuoted 16 lines · Greg Kroah-Hartman
Quoted 9 lines · Andreas
Quoted 8 lines
So, sorry, I have to stop here at this series. I've applied the first 3 to the opw-next branch of staging.git so they can live somewhere until 3.13-rc1 is out.
I know you spent a lot of time making these 120 patches to send me, but that too is crazy. You shouldn't wait that long to get feedback and send patches to me at all. Please send them in smaller series, with less time between patch submissions.
Quoted 5 lines
The reason that there are so many patches in a burst is that we are also developing new features and fixing bugs in parallel with the kernel, but they also need to be tested a lot before they are released. It's not any different from kernel patches testing on -next or -mm for a few months before they are pushed to the mainline kernel in a batch.
Quoted 9 lines
It's very different in that you are expecting me to suddenly accept this huge burst of patches all at once, and I can't review them properly. As this patch series shows, there was a problem in the 4rd patch in a 100+ series, which means that I couldn't take them all. So you wasted a bunch of work preparing those other 96 patches.
Send them to me in short chunks, that way you don't waste time, and you don't take so long between patch acceptance.
working on it. sorry for the noise.
Quoted 6 lines
Quoted 5 lines
The out-of-tree development can't really stop for a year while the kernel client code is cleaned up. If the feature patches don't land into the kernel client for a year (or however long it takes to do all the cleanup), then it will become outdated and the whole reason for adding the client into the kernel is lost.
Quoted 9 lines
But you understand my hesitation at taking any new features when there really has not been any attempt by your team to do much cleanup and fixes of the code at all, right? It feels like I have done more cleanup personally than anyone on your teams, is this not true?
So, why would I believe that you all are really going to start doing the major cleanup work on this code that is so obviously needed? Why would I take new features that you are spending your time on instead?
My apologize. I was distracted by other projects for some time and I am trying to make it up. I thought you agreed that I can first close the patch gap between in-kernel client and external tree, and then add cleanup patches, no?
Quoted 16 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
There are many users of the external tree so we cannot just abandon it, especially that the upstream client is not shipped in any distribution yet. Thanks for your understanding.
What is keeping people using that tree? Support for older/distro kernels?
Quoted 7 lines
Support for distro kernels is a big part of it. Most HPC sites use vendor kernels of various vintages, and we need to keep the code working for those sites. We also need to continue developing features needed by ever-larger clusters, fixing bugs, etc. Eventually, when Lustre is in the kernel proper,
it will also be available in future distro kernels and we can eventually stop supporting the out-of-tree code. I expect that to be 3+ years away.
Quoted 8 lines
So, originally you said it would take about a year to get this out of staging. It's been 6 months already with no real progress made with the exception of having interns do cleanup and me doing some basic wrapper function removals. What's the plan for the next 6 months? Based on these 100+ patches, I don't see any major changes that are happening to get this cleaned up properly (or did I miss those patches in this huge patchbomb?)
sorry for sending you these patch bombs. I wanted to first sync with external tree because otherwise the gap will be increasingly large and eventually we end up with two different code base.
Quoted 28 lines
Quoted 21 lines
Quoted 1 line
Is it the fact that the server code isn't in the kernel?
Not really. Lustre servers are on separate servers with vendor kernels (ancient by your standards), and there isn't really any demand for using newer kernels on those nodes. Most importantly, the out-of-tree branch is where all of the new feature development is being done. That also means if feature patches don't land into the kernel it just makes the kernel version less attractive for users.
Quoted 5 lines
Should that be added now too so that we can get a proper view of what
can and can not be changed? Some of your patches are showing that things are shared by the two chunks of code, so does that mean if I delete things in the client code that don't look to be used by anything, you will have problems because the server now breaks?Adding the server code to staging would mean another 150kLOC in staging. We also haven't cleaned that code up even nearly as much as the client code, so it isn't really even ready to go into staging either. I don't think you or anyone else would be happy to see that code yet, so I don't think there is a real advantage to do so.
Quoted 6 lines
Deleting unused code in the kernel client isn't fatal, since we'll still have the out-of-tree version, but we're trying to keep the code in sync as much as possible so that it is easier to port patches in both directions. If code is deleted it means more that needs to be added back later when the server eventually does get merged, and more effort to resolve patch conflicts.
Quoted 7 lines
But look at the function that caused this original question to be asked. You want an api change that I would never accept as it doesn't do anything to the client code at all. How do I "know" this is acceptable and needed if I can't see the server side? How would I "know" not to take a cleanup patch that reverted this change if I can't see the server side at the same time?
How about CCing Andreas and me for patches changing drivers/staging/lustre? If such reverting patch appears, we can send NACKs, right? Andreas reviews most of patches landed in both in kernel client and in the external tree. He for sure knows if a cleanup patch drops something that server needs. Also I'll try to add comments on things that are used only by server but client also needs. Does it sound working?
Quoted 5 lines
150k of code is not a big deal to me, I can easily take it. If it means that we have a real chance at getting this all fixed up properly, I say to do it, otherwise I really don't think this project will ever succeed, sorry.
The issue with server code is that it is not even updated to support latest kernel. Also work is still going on to get rid of extra patches to generic kernel code. And there is the issue of ldiskfs that is mostly the same code as ext4 but with modifications. I am not even sure if Ted is willing to take these extra lustre patches. I mean, if they can be merged, they should have been sent upstream and merged long ago. Is it correct, Andreas?
Thanks,
Tao
Greg Kroah-Hartman
maintainerRe: [PATCH 05/40] staging/lustre: validate open handle cookies
In reply to Peng TaoQuoted 48 lines · Peng Tao
Quoted 46 lines · Greg Kroah-Hartman
Quoted 17 lines · Andreas
Quoted 7 lines · Greg Kroah-Hartman
Quoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.Quoted 6 lines
Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
Quoted 6 lines
The
portals_handleis used as a generic type for objects referenced over the network, like a file handle. The "owner" parameter is just used as an extra check that the cookie passed from the client is actually a valid value for the code in which it is being used (e.g. metadata or data object). It isn't actually dereferenced by anything, it is just like a magic value.Then make it an explicit type, not a void *.
Quoted 18 lines
Quoted 10 lines
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.Quoted 3 lines
This changelog entry doesn't even match up with the code below. ALl callers of
class_handle2objectare passing NULL here, which makes this patch pretty pointless, right?As Tao wrote, this is the patch summary that matches what was committed in our own tree and in this case mostly describes the changes made on the server. Keeping the same commits and comments in both trees makes it easier to keep the code in sync.
Quoted 6 lines
Quoted 4 lines
Ok, but as it is, this patch does nothing to the client code, so how can I accept it? A function that is only ever called with NULL as an option is ripe for cleanup in my eyes.
How about adding a comment above the function to note that this extra argument is used by server code and please don't remove it?
How about adding the server code to the kernel to keep problems like this (which will continue to crop up, it's not just this one function, right?) from happening in the future?
In-kernel code does not depend on out-of-kernel code, it's that simple, and has been a rule for kernel code for forever. Either deal with the fact that you will have to keep the apis and functions working for your out-of-tree code, or put all the code into the kernel. Don't force in-kernel code to deal with out-of-tree code as there is NO way that anyone other than the very few of you, can deal with that at all.
greg k-h
Greg Kroah-Hartman
maintainerRe: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
In reply to Peng TaoQuoted 40 lines · Peng Tao
Quoted 16 lines · Greg Kroah-Hartman
Quoted 9 lines · Andreas
Quoted 8 lines
So, sorry, I have to stop here at this series. I've applied the first 3 to the opw-next branch of staging.git so they can live somewhere until 3.13-rc1 is out.
I know you spent a lot of time making these 120 patches to send me, but that too is crazy. You shouldn't wait that long to get feedback and send patches to me at all. Please send them in smaller series, with less time between patch submissions.
Quoted 5 lines
The reason that there are so many patches in a burst is that we are also developing new features and fixing bugs in parallel with the kernel, but they also need to be tested a lot before they are released. It's not any different from kernel patches testing on -next or -mm for a few months before they are pushed to the mainline kernel in a batch.
Quoted 9 lines
It's very different in that you are expecting me to suddenly accept this huge burst of patches all at once, and I can't review them properly. As this patch series shows, there was a problem in the 4rd patch in a 100+ series, which means that I couldn't take them all. So you wasted a bunch of work preparing those other 96 patches.
Send them to me in short chunks, that way you don't waste time, and you don't take so long between patch acceptance.
working on it. sorry for the noise.
Quoted 11 lines
Quoted 5 lines
The out-of-tree development can't really stop for a year while the kernel client code is cleaned up. If the feature patches don't land into the kernel client for a year (or however long it takes to do all the cleanup), then it will become outdated and the whole reason for adding the client into the kernel is lost.
But you understand my hesitation at taking any new features when there really has not been any attempt by your team to do much cleanup and fixes of the code at all, right? It feels like I have done more cleanup personally than anyone on your teams, is this not true?
Quoted 8 lines
Quoted 4 lines
So, why would I believe that you all are really going to start doing the major cleanup work on this code that is so obviously needed? Why would I take new features that you are spending your time on instead?
My apologize. I was distracted by other projects for some time and I am trying to make it up. I thought you agreed that I can first close the patch gap between in-kernel client and external tree, and then add cleanup patches, no?
What exactly is "the gap"? Are we talking 200 patches? 10? 100? I have no idea what you are referring to, but realize I can't review 100 patches at a single time easily.
And what happens when I accep these "gap" patches? Will development in this external tree suddenly stop? It sure doesn't seem like this is happening as I thought it would have already since the code has been in the kernel for over 6 months now.
Quoted 24 lines
Quoted 16 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
There are many users of the external tree so we cannot just abandon it, especially that the upstream client is not shipped in any distribution yet. Thanks for your understanding.
What is keeping people using that tree? Support for older/distro kernels?
Quoted 7 lines
Support for distro kernels is a big part of it. Most HPC sites use vendor kernels of various vintages, and we need to keep the code working for those sites. We also need to continue developing features needed by ever-larger clusters, fixing bugs, etc. Eventually, when Lustre is in the kernel proper,
it will also be available in future distro kernels and we can eventually stop supporting the out-of-tree code. I expect that to be 3+ years away.Quoted 8 lines
So, originally you said it would take about a year to get this out of staging. It's been 6 months already with no real progress made with the exception of having interns do cleanup and me doing some basic wrapper function removals. What's the plan for the next 6 months? Based on these 100+ patches, I don't see any major changes that are happening to get this cleaned up properly (or did I miss those patches in this huge patchbomb?)
Quoted 3 lines
sorry for sending you these patch bombs. I wanted to first sync with external tree because otherwise the gap will be increasingly large and eventually we end up with two different code base.
You already have 2 different code bases :(
Quoted 35 lines
Quoted 28 lines
Quoted 21 lines
Quoted 1 line
Is it the fact that the server code isn't in the kernel?
Not really. Lustre servers are on separate servers with vendor kernels (ancient by your standards), and there isn't really any demand for using newer kernels on those nodes. Most importantly, the out-of-tree branch is where all of the new feature development is being done. That also means if feature patches don't land into the kernel it just makes the kernel version less attractive for users.
Quoted 5 lines
Should that be added now too so that we can get a proper view of what
can and can not be changed? Some of your patches are showing that things are shared by the two chunks of code, so does that mean if I delete things in the client code that don't look to be used by anything, you will have problems because the server now breaks?Adding the server code to staging would mean another 150kLOC in staging. We also haven't cleaned that code up even nearly as much as the client code, so it isn't really even ready to go into staging either. I don't think you or anyone else would be happy to see that code yet, so I don't think there is a real advantage to do so.
Quoted 6 lines
Deleting unused code in the kernel client isn't fatal, since we'll still have the out-of-tree version, but we're trying to keep the code in sync as much as possible so that it is easier to port patches in both directions. If code is deleted it means more that needs to be added back later when the server eventually does get merged, and more effort to resolve patch conflicts.
Quoted 7 lines
But look at the function that caused this original question to be asked. You want an api change that I would never accept as it doesn't do anything to the client code at all. How do I "know" this is acceptable and needed if I can't see the server side? How would I "know" not to take a cleanup patch that reverted this change if I can't see the server side at the same time?
Quoted 2 lines
How about CCing Andreas and me for patches changing
drivers/staging/lustre?
I have been trying to do that, but not all developers follow that. Also, not all developers in the overall kernel do that either, for good reasons (i.e. api changes.)
Quoted 1 line
If such reverting patch appears, we can send NACKs, right?
Not always, no. Being a maintainer doesn't mean that you have to have final approval on every change to your codebase, this is a long-time thing you all should know quite well.
Quoted 5 lines
Andreas reviews most of patches landed in both in kernel client and in the external tree. He for sure knows if a cleanup patch drops something that server needs. Also I'll try to add comments on things that are used only by server but client also needs. Does it sound working?
Does the current way of working seem to work for you?
Quoted 7 lines
Quoted 5 lines
150k of code is not a big deal to me, I can easily take it. If it means that we have a real chance at getting this all fixed up properly, I say to do it, otherwise I really don't think this project will ever succeed, sorry.
The issue with server code is that it is not even updated to support latest kernel.
Then how are you even testing this stuff?
greg k-h
Re: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
[CC Sorin since he might be interested in such discussion.]
Quoted 45 lines · Greg Kroah-Hartman
Quoted 44 lines · Peng Tao
Quoted 16 lines · Greg Kroah-Hartman
Quoted 9 lines · Andreas
Quoted 8 lines
So, sorry, I have to stop here at this series. I've applied the first 3 to the opw-next branch of staging.git so they can live somewhere until 3.13-rc1 is out.
I know you spent a lot of time making these 120 patches to send me, but that too is crazy. You shouldn't wait that long to get feedback and send patches to me at all. Please send them in smaller series, with less time between patch submissions.
Quoted 5 lines
The reason that there are so many patches in a burst is that we are also developing new features and fixing bugs in parallel with the kernel, but they also need to be tested a lot before they are released. It's not any different from kernel patches testing on -next or -mm for a few months before they are pushed to the mainline kernel in a batch.
Quoted 9 lines
It's very different in that you are expecting me to suddenly accept this huge burst of patches all at once, and I can't review them properly. As this patch series shows, there was a problem in the 4rd patch in a 100+ series, which means that I couldn't take them all. So you wasted a bunch of work preparing those other 96 patches.
Send them to me in short chunks, that way you don't waste time, and you don't take so long between patch acceptance.
working on it. sorry for the noise.
Quoted 15 lines
Quoted 5 lines
The out-of-tree development can't really stop for a year while the kernel client code is cleaned up. If the feature patches don't land into the kernel client for a year (or however long it takes to do all the cleanup), then it will become outdated and the whole reason for adding the client into the kernel is lost.
But you understand my hesitation at taking any new features when there really has not been any attempt by your team to do much cleanup and fixes of the code at all, right? It feels like I have done more cleanup personally than anyone on your teams, is this not true?
So, why would I believe that you all are really going to start doing the major cleanup work on this code that is so obviously needed? Why would I take new features that you are spending your time on instead?
Quoted 9 lines
Quoted 4 lines
My apologize. I was distracted by other projects for some time and I am trying to make it up. I thought you agreed that I can first close the patch gap between in-kernel client and external tree, and then add cleanup patches, no?
What exactly is "the gap"? Are we talking 200 patches? 10? 100? I have no idea what you are referring to, but realize I can't review 100 patches at a single time easily.
Right now, I think it is about 150 patches. I will split them into smaller chunks (10 patches or in each patchset) and send them incrementally.
Quoted 5 lines
And what happens when I accep these "gap" patches? Will development in this external tree suddenly stop? It sure doesn't seem like this is happening as I thought it would have already since the code has been in the kernel for over 6 months now.
No, it won't stop. The biggest feature coming in this time is HSM. In future, I think there will be less features and more bug fixes. Porting patches from external tree isn't just to make the two tree in sync. External tree contains many bug fix patches that in kernel client wants as well. If we stop porting, as time goes by, the in tree client will be less featured, and more buggy than external tree. When we get it out of staging like that, it will be very hard to persuade people to use it.
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
There are many users of the external tree so we cannot just abandon it, especially that the upstream client is not shipped in any distribution yet. Thanks for your understanding.
What is keeping people using that tree? Support for older/distro kernels?
Quoted 7 lines
Support for distro kernels is a big part of it. Most HPC sites use vendor kernels of various vintages, and we need to keep the code working for those sites. We also need to continue developing features needed by ever-larger clusters, fixing bugs, etc. Eventually, when Lustre is in the kernel proper,
it will also be available in future distro kernels and we can eventually stop supporting the out-of-tree code. I expect that to be 3+ years away.So, originally you said it would take about a year to get this out of staging. It's been 6 months already with no real progress made with the exception of having interns do cleanup and me doing some basic wrapper function removals. What's the plan for the next 6 months? Based on these 100+ patches, I don't see any major changes that are happening to get this cleaned up properly (or did I miss those patches in this huge patchbomb?)
Quoted 5 lines
Quoted 3 lines
sorry for sending you these patch bombs. I wanted to first sync with external tree because otherwise the gap will be increasingly large and eventually we end up with two different code base.
You already have 2 different code bases :(
we still want to make the difference as small as possible. So it won't cost too much effort to port patches between them.
Quoted 43 lines
Quoted 36 lines
Quoted 29 lines
Quoted 22 lines
Quoted 2 lines
Is it the fact that the server code isn't in the kernel?
Not really. Lustre servers are on separate servers with vendor kernels (ancient by your standards), and there isn't really any demand for using newer kernels on those nodes. Most importantly, the out-of-tree branch is where all of the new feature development is being done. That also means if feature patches don't land into the kernel it just makes the kernel version less attractive for users.
Quoted 5 lines
Should that be added now too so that we can get a proper view of what
can and can not be changed? Some of your patches are showing that things are shared by the two chunks of code, so does that mean if I delete things in the client code that don't look to be used by anything, you will have problems because the server now breaks?Adding the server code to staging would mean another 150kLOC in staging. We also haven't cleaned that code up even nearly as much as the client code, so it isn't really even ready to go into staging either. I don't think you or anyone else would be happy to see that code yet, so I don't think there is a real advantage to do so.
Quoted 6 lines
Deleting unused code in the kernel client isn't fatal, since we'll still have the out-of-tree version, but we're trying to keep the code in sync as much as possible so that it is easier to port patches in both directions. If code is deleted it means more that needs to be added back later when the server eventually does get merged, and more effort to resolve patch conflicts.
Quoted 7 lines
But look at the function that caused this original question to be asked. You want an api change that I would never accept as it doesn't do anything to the client code at all. How do I "know" this is acceptable and needed if I can't see the server side? How would I "know" not to take a cleanup patch that reverted this change if I can't see the server side at the same time?
Quoted 2 lines
How about CCing Andreas and me for patches changing
drivers/staging/lustre?I have been trying to do that, but not all developers follow that. Also, not all developers in the overall kernel do that either, for good reasons (i.e. api changes.)
Quoted 5 lines
Quoted 1 line
If such reverting patch appears, we can send NACKs, right?
Not always, no. Being a maintainer doesn't mean that you have to have final approval on every change to your codebase, this is a long-time thing you all should know quite well.
I see that we don't have final approval. But at least we can send review comments and raise objections, right? That's my understanding of how kernel development on public mailing list works, every one gets the chance to voice his/her opinions. :)
Quoted 9 lines
Quoted 6 lines
Andreas reviews most of patches landed in both in kernel client and in the external tree. He for sure knows if a cleanup patch drops something that server needs. Also I'll try to add comments on things that are used only by server but client also needs. Does it sound working?
Does the current way of working seem to work for you?
I think it is working but needs improvement. All my patches were sent to Andreas for inspection before sending out public. We might need to speed it up. Andreas, what do you think of that I just send patches public and you do the inspection on mailing list? That way Greg and others can review and comment at the same time.
After closing the current patch gap, I can work on cleaning up and most cleanups are quite obvious.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
150k of code is not a big deal to me, I can easily take it. If it means that we have a real chance at getting this all fixed up properly, I say to do it, otherwise I really don't think this project will ever succeed, sorry.
The issue with server code is that it is not even updated to support latest kernel.
Then how are you even testing this stuff?
We can set up Lustre server with supported vendor kernels. For details if you are interested, please see
https://wiki.hpdd.intel.com/display/PUB/Putting+together+a+Lustre+filesystem
It is quite common that Lustre client and server use different kernel versions. Since we only have client code in kernel, we can always test latest kernel client against a running Lustre server regardless of the server kernel version.
Thanks,
Tao
Re: [PATCH 05/40] staging/lustre: validate open handle cookies
In reply to Greg Kroah-HartmanQuoted 53 lines · Greg Kroah-Hartman
Quoted 52 lines · Peng Tao
Quoted 50 lines · Greg Kroah-Hartman
Quoted 17 lines · Andreas
Quoted 7 lines · Greg Kroah-Hartman
Quoted 5 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add a const void *owner parameter toclass_handle2object()which must be matched by theh_ownermember of the handle in addition to the cookie.Quoted 6 lines
Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle", never. This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can catch our problems, and don't try to cheat by using void *.
Quoted 6 lines
The
portals_handleis used as a generic type for objects referenced over the network, like a file handle. The "owner" parameter is just used as an extra check that the cookie passed from the client is actually a valid value for the code in which it is being used (e.g. metadata or data object). It isn't actually dereferenced by anything, it is just like a magic value.Then make it an explicit type, not a void *.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL as the argument to the owner parameter, except in the case ofmdt_handle2mfd()where we add an explicitmdt_export_dataparameter which we use as the owner when searching for a MFD. When allocating a new MFD, pass a pointer to themdt_export_dataintomdt_mfd_new()and store it inh_owner. This allows the MDT to validate that the client has not sent the wrong open handle cookie, or sent the right cookie to the wrong MDT.Quoted 8 lines
Quoted 3 lines
This changelog entry doesn't even match up with the code below. ALl callers of
class_handle2objectare passing NULL here, which makes this patch pretty pointless, right?As Tao wrote, this is the patch summary that matches what was committed in our own tree and in this case mostly describes the changes made on the server. Keeping the same commits and comments in both trees makes it easier to keep the code in sync.
Ok, but as it is, this patch does nothing to the client code, so how can I accept it? A function that is only ever called with NULL as an option is ripe for cleanup in my eyes.
Quoted 7 lines
Quoted 2 lines
How about adding a comment above the function to note that this extra argument is used by server code and please don't remove it?
How about adding the server code to the kernel to keep problems like this (which will continue to crop up, it's not just this one function, right?) from happening in the future?
As explained in the other thread, the server code is not even ready for landing in upstream kernel. And it won't be for quite some time.
Quoted 7 lines
In-kernel code does not depend on out-of-kernel code, it's that simple, and has been a rule for kernel code for forever. Either deal with the fact that you will have to keep the apis and functions working for your out-of-tree code, or put all the code into the kernel. Don't force in-kernel code to deal with out-of-tree code as there is NO way that anyone other than the very few of you, can deal with that at all.
Fair enough. Andreas, how about we handling this kind of difference in external tree and letting in-tree client be clean of it? We already have HAVE_SERVER_SUPPORT macro in external tree. It is just a matter of adding more references.
Thanks,
Tao
Re: [PATCH 05/40] staging/lustre: validate open handle cookies
Quoted 93 lines · Peng Tao
Quoted 72 lines · Greg Kroah-Hartman
Quoted 71 lines · Peng Tao
Quoted 2 lines · Greg Kroah-Hartman
Quoted 1 line · Andreas
On 2013/11/14 9:13 PM, "Greg Kroah-Hartman"
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines · wrote:
Quoted 3 lines · Peng Tao
From: "John L. Hammond" <[email protected]>
Add a const void
*h_ownermember to structportals_handle. Add aconst
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
void *owner parameter to
class_handle2object()which must bematched
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
by the
h_ownermember of the handle in addition to the cookie.Ick ick ick.
NEVER use a void pointer if you can help it, and for a "handle",
never.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This isn't other operating systems, sorry. We know what types our pointers to structures are, use them, so that the compiler can
catch our
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
problems, and don't try to cheat by using void *.
The
portals_handleis used as a generic type for objects referencedover
Quoted 1 line
Quoted 1 line
the network, like a file handle. The "owner" parameter is just
used as
Quoted 3 lines
Quoted 3 lines
an extra check that the cookie passed from the client is actually a valid value for the code in which it is being used (e.g. metadata or data object). It isn't actually dereferenced by anything, it is
just
Quoted 6 lines
Quoted 1 line
like a magic value.
Then make it an explicit type, not a void *.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Adjust
the callers ofclass_handle2object()accordingly, using NULL asthe
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
argument to the owner parameter, except in the case of
mdt_handle2mfd()where we add an explicitmdt_export_dataparameter
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
which we use as the owner when searching for a MFD. When
allocating a
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
new MFD, pass a pointer to the
mdt_export_datainto
mdt_mfd_new()andQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
store it in
h_owner. This allows the MDT to validate that theclient
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
has not sent the wrong open handle cookie, or sent the right
cookie to
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the wrong MDT.
This changelog entry doesn't even match up with the code below.
ALl
Quoted 1 line
Quoted 1 line
Quoted 1 line
callers of
class_handle2objectare passing NULL here, which makesthis
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
patch pretty pointless, right?
As Tao wrote, this is the patch summary that matches what was
committed
Quoted 1 line
Quoted 1 line
in our own tree and in this case mostly describes the changes made
on the
Quoted 1 line
Quoted 1 line
server. Keeping the same commits and comments in both trees makes
it
Quoted 3 lines
Quoted 1 line
easier to keep the code in sync.
Ok, but as it is, this patch does nothing to the client code, so how
can
Quoted 1 line
I accept it? A function that is only ever called with NULL as an
option
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
is ripe for cleanup in my eyes.
How about adding a comment above the function to note that this extra argument is used by server code and please don't remove it?
How about adding the server code to the kernel to keep problems like this (which will continue to crop up, it's not just this one function, right?) from happening in the future?
As explained in the other thread, the server code is not even ready for landing in upstream kernel. And it won't be for quite some time.
Quoted 7 lines
In-kernel code does not depend on out-of-kernel code, it's that simple, and has been a rule for kernel code for forever. Either deal with the fact that you will have to keep the apis and functions working for your out-of-tree code, or put all the code into the kernel. Don't force in-kernel code to deal with out-of-tree code as there is NO way that anyone other than the very few of you, can deal with that at all.
Quoted 4 lines
Fair enough. Andreas, how about we handling this kind of difference in external tree and letting in-tree client be clean of it? We already have
HAVE_SERVER_SUPPORTmacro in external tree. It is just a matter of adding more references.
Fine. This patch is not critical, and the API isn't used very widely in the code, so it shouldn't cause too much grief.
Cheers, Andreas
Signature · 4 lines
Andreas Dilger Lustre Software Architect Intel High Performance Data Division
Greg Kroah-Hartman
maintainerRe: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
In reply to Peng TaoQuoted 9 lines · Peng Tao
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
So, why would I believe that you all are really going to start doing the major cleanup work on this code that is so obviously needed? Why would I take new features that you are spending your time on instead?
My apologize. I was distracted by other projects for some time and I am trying to make it up. I thought you agreed that I can first close the patch gap between in-kernel client and external tree, and then add cleanup patches, no?
Quoted 7 lines
Quoted 4 lines
What exactly is "the gap"? Are we talking 200 patches? 10? 100? I have no idea what you are referring to, but realize I can't review 100 patches at a single time easily.
Right now, I think it is about 150 patches. I will split them into smaller chunks (10 patches or in each patchset) and send them incrementally.
Good.
Quoted 5 lines
Quoted 5 lines
And what happens when I accep these "gap" patches? Will development in this external tree suddenly stop? It sure doesn't seem like this is happening as I thought it would have already since the code has been in the kernel for over 6 months now.
Quoted 8 lines
No, it won't stop. The biggest feature coming in this time is HSM. In future, I think there will be less features and more bug fixes. Porting patches from external tree isn't just to make the two tree in sync. External tree contains many bug fix patches that in kernel client wants as well. If we stop porting, as time goes by, the in tree client will be less featured, and more buggy than external tree. When we get it out of staging like that, it will be very hard to persuade people to use it.
So, who is going to take the time to do the cleanup and work involved in getting this into mergable state, if developers are off adding new features to your out-of-tree kernel code?
I _really_ hate taking new features for staging drivers, as that shows that the developers are using staging as just a dumping ground to get stuff merged, without taking the time to get it cleaned up properly.
If this continues, and no major work is done to clean things up within the next 6 months (i.e. the same thing as the past 6 months.) this code will be deleted from the tree. You have been warned.
Quoted 3 lines
Quoted 1 line
You already have 2 different code bases :(
we still want to make the difference as small as possible. So it won't cost too much effort to port patches between them.
That is your problem, not ours, sorry.
Best of luck,
greg k-h
Re: [PATCH 04/40] staging/lustre/llite: Access to released file trigs a restore
Quoted 34 lines · Greg Kroah-Hartman
Quoted 9 lines · Peng Tao
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
So, why would I believe that you all are really going to start doing the major cleanup work on this code that is so obviously needed? Why would I take new features that you are spending your time on instead?
My apologize. I was distracted by other projects for some time and I am trying to make it up. I thought you agreed that I can first close the patch gap between in-kernel client and external tree, and then add cleanup patches, no?
Quoted 7 lines
Quoted 4 lines
What exactly is "the gap"? Are we talking 200 patches? 10? 100? I have no idea what you are referring to, but realize I can't review 100 patches at a single time easily.
Right now, I think it is about 150 patches. I will split them into smaller chunks (10 patches or in each patchset) and send them incrementally.
Good.
Quoted 5 lines
Quoted 5 lines
And what happens when I accep these "gap" patches? Will development in this external tree suddenly stop? It sure doesn't seem like this is happening as I thought it would have already since the code has been in the kernel for over 6 months now.
Quoted 8 lines
No, it won't stop. The biggest feature coming in this time is HSM. In future, I think there will be less features and more bug fixes. Porting patches from external tree isn't just to make the two tree in sync. External tree contains many bug fix patches that in kernel client wants as well. If we stop porting, as time goes by, the in tree client will be less featured, and more buggy than external tree. When we get it out of staging like that, it will be very hard to persuade people to use it.
Quoted 4 lines
So, who is going to take the time to do the cleanup and work involved in getting this into mergable state, if developers are off adding new features to your out-of-tree kernel code?
I am to work on it.
Quoted 8 lines
I _really_ hate taking new features for staging drivers, as that shows that the developers are using staging as just a dumping ground to get stuff merged, without taking the time to get it cleaned up properly.
If this continues, and no major work is done to clean things up within the next 6 months (i.e. the same thing as the past 6 months.) this code will be deleted from the tree. You have been warned.
Got it.
Thanks,
Tao