osd: speedup startup by finishing pending removals in background
PATCH 1/1 osd: speedup startup by finishing pending removals in background
This patch applies on top of the one I just posted, Subject “osd: avoid flushing every TEMP removal to speedup startup”.
When PG removals are underway and the OSD is restarted, or when multiple removals are scheduled manually with ceph_filestore_dump premove, the OSD may take a long time to process all pending removals before it will join the cluster.
This patch introduces an option that enables the OSD to join the cluster first, performing the removals in background while actively participating in the cluster, as the OSD would if it hadn't been restarted.
In hindsight, I suppose it might have been wiser to add a data member to DeletingStateRef to hold the coll_t, instead of having to search for it again, but this patch is what I tested, and it's likely good enough for now.
--- src/common/config_opts.h | 10 ++++++ src/osd/OSD.cc | 75 ++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 82 insertions(+), 3 deletions(-)
src/common/config_opts.h +10 -0
@@ -583,6 +583,16 @@ OPTION(osd_client_op_priority, OPT_U32, 63) OPTION(osd_recovery_op_priority, OPT_U32, 10)OPTION (osd_recovery_op_warn_multiple,OPT_U32 ,16 ) +// Removal of PGs is done in background, but if the osd is restarted, +// it will finish all pending removals before joining the cluster. +// This can take a while. If this option is set to true, then pending +// removals will be performed in background, while the osd runs +// normally. This is a bit dangerous if the OSD gets a new copy of +// the PG before the pending removal is completed: attributes stored +// in the leveldb may be lost when removal cleans up an object's +// attributes AFTER the new object is backfilled. +OPTION (osd_startup_finish_remove_in_background,OPT_BOOL , false) + // Max time to wait between notifying mon of shutdown and shutting downOPTION (osd_mon_shutdown_timeout,OPT_DOUBLE ,5 )
src/osd/OSD.cc +72 -3
110 lines not on this page · raw message
Signature · 8 lines
Alexandre Oliva, freedom fighter http://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Alexandre Oliva
In hindsight, I suppose it might have been wiser to add a data member to DeletingStateRef to hold the
coll_t, instead of having to search for it again, but this patch is what I tested, and it's likely good enough for now.
I've now implemented this, and verified that it works.
I've also arranged for attempts to resurrect a pending removal from an earlier session to be delayed rather than to cause an abort. I haven't tested this scenario, though; it might be the case that the osd would timeout and suicide or somesuch.
---
When PG removals are underway and the OSD is restarted, or when multiple removals are scheduled manually with ceph_filestore_dump premove, the OSD may take a long time to process all pending removals before it will join the cluster.
This patch introduces an option that enables the OSD to join the cluster first, performing the removals in background while actively participating in the cluster, as the OSD would if it hadn't been restarted.
--- src/common/config_opts.h | 10 +++++++++ src/osd/OSD.cc | 52 +++++++++++++++++++++++++++++++++++++++++++--- src/osd/OSD.h | 17 +++++++++++++-- 3 files changed, 74 insertions(+), 5 deletions(-)
src/common/config_opts.h +10 -0
@@ -583,6 +583,16 @@ OPTION(osd_client_op_priority, OPT_U32, 63) OPTION(osd_recovery_op_priority, OPT_U32, 10)OPTION (osd_recovery_op_warn_multiple,OPT_U32 ,16 ) +// Removal of PGs is done in background, but if the osd is restarted, +// it will finish all pending removals before joining the cluster. +// This can take a while. If this option is set to true, then pending +// removals will be performed in background, while the osd runs +// normally. This is a bit dangerous if the OSD gets a new copy of +// the PG before the pending removal is completed: attributes stored +// in the leveldb may be lost when removal cleans up an object's +// attributes AFTER the new object is backfilled. +OPTION (osd_startup_finish_remove_in_background,OPT_BOOL , false) + // Max time to wait between notifying mon of shutdown and shutting downOPTION (osd_mon_shutdown_timeout,OPT_DOUBLE ,5 )
src/osd/OSD.cc +49 -3
81 lines not on this page · raw message
src/osd/OSD.h +15 -2
@@ -217,9 +217,21 @@ class DeletingState { public:const spg_t pgid;const PGRef old_pg_state; +const coll_t old_coll;// iff old_pg_state is NULL DeletingState(const pair<spg_t, PGRef> &in) : lock("DeletingState::lock"), status(QUEUED), stop_deleting(false), - pgid(in.first), old_pg_state(in.second) {} + pgid(in.first), old_pg_state(in.second), old_coll() {} + DeletingState(const pair<spg_t, coll_t> &in) : + lock("DeletingState::lock"), status(QUEUED), stop_deleting(false), + pgid(in.first), old_pg_state(NULL), old_coll(in.second) {} + + bool resurrectable_p() const { +return !!old_pg_state; + } + coll_t get_coll() const { +assert (!resurrectable_p ()); +return old_coll; + }/// transition status to clearing bool start_clearing () {
@@ -286,7 +298,8 @@ public:/// try to halt the deletion bool try_stop_deletion () { Mutex::Locker l(lock); - stop_deleting = true; +if (resurrectable_p ()) + stop_deleting = true;/** * If we are in DELETING_DIR or CLEARING_DIR, there are in progress * operations we have to wait for before continuing on. States
Signature · 4 lines
Alexandre Oliva, freedom fighter http://FSFLA.org/~lxoliva/ You must be the change you wish to see in the world. -- Gandhi Be Free! -- http://FSFLA.org/ FSF Latin America board member Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer