Skip to contentSkip to conversation

gfs2: misc recovery patch collection

PATCH Superseded Bob Peterson 3 participants 24 messages Apr 30, 2019 – May 7, 2019 GFS2 file system

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover gfs2: misc recovery patch collection

Here is version 3 of the patch set I posted on 23 April. It is revised based on bugs I found testing with xfstests.

This is a collection of patches I've been using to address the myriad of recovery problems I've found. I'm still finding them, so the battle is not won yet. I'm not convinced we need all of these but I thought I'd send them anyway and get feedback. Previously I sent out a version of the patch "gfs2: Force withdraw to replay journals and wait for it to finish" that was too big and complex. So I broke it up into four patches, starting with "move check_journal_clean to util.c for future use". So those four need to be a set. There aren't many other dependencies between patches, so the others could probably be taken or rejected individually. There are more patches I still need to perfect, but maybe a few of the safer ones can be pushed to for-next.

Bob Peterson (19):
gfs2: kthread and remount improvements
gfs2: eliminate tr_num_revoke_rm
gfs2: log which portion of the journal is replayed
gfs2: Warn when a journal replay overwrites a rgrp with buffers
gfs2: Introduce concept of a pending withdraw
gfs2: log error reform
gfs2: Only complain the first time an io error occurs in quota or log
gfs2: Stop ail1 wait loop when withdrawn
gfs2: Ignore recovery attempts if gfs2 has io error or is withdrawn
gfs2: move check_journal_clean to util.c for future use
gfs2: Allow some glocks to be used during withdraw
gfs2: Don't loop forever in gfs2_freeze if withdrawn
gfs2: Make secondary withdrawers wait for first withdrawer
gfs2: Don't write log headers after file system withdraw
gfs2: Force withdraw to replay journals and wait for it to finish
gfs2: simply gfs2_freeze by removing case
gfs2: Add verbose option to check_journal_clean
gfs2: Check for log write errors before telling dlm to unlock
gfs2: Do log_flush in gfs2_ail_empty_gl even if ail list is empty

 fs/gfs2/aops.c       |   4 +-
 fs/gfs2/file.c       |   2 +-
 fs/gfs2/glock.c      |  82 +++++++++++++++--
 fs/gfs2/glock.h      |   1 +
 fs/gfs2/glops.c      | 100 +++++++++++++++++++--
 fs/gfs2/glops.h      |   3 +-
 fs/gfs2/incore.h     |  21 ++++-
 fs/gfs2/inode.c      |  14 ++-
 fs/gfs2/lock_dlm.c   |  50 +++++++++++
 fs/gfs2/log.c        |  55 +++++++-----
 fs/gfs2/log.h        |   1 +
 fs/gfs2/lops.c       |  27 +++++-
 fs/gfs2/meta_io.c    |   6 +-
 fs/gfs2/ops_fstype.c |  65 ++++----------
 fs/gfs2/quota.c      |  10 ++-
 fs/gfs2/recovery.c   |   3 +-
 fs/gfs2/super.c      |  88 +++++++++++-------
 fs/gfs2/super.h      |   1 +
 fs/gfs2/sys.c        |  14 ++-
 fs/gfs2/trans.c      |   6 +-
 fs/gfs2/util.c       | 206 ++++++++++++++++++++++++++++++++++++++++---
 fs/gfs2/util.h       |  15 ++++
 22 files changed, 617 insertions(+), 157 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v3 1/19 gfs2: kthread and remount improvements

Before this patch, gfs2 saved the pointers to the two daemon threads (logd and quotad) in the superblock, but they were never cleared, even if the threads were stopped (e.g. on remount -o ro). That meant that certain error conditions (like a withdrawn file system) could race. For example, xfstests generic/361 caused an IO error during remount -o ro, which caused the kthreads to be stopped, then the error flagged. Later, when the test unmounted the file system, it would try to stop the threads a second time with kthread_stop.

This patch does two things: First, every time it stops the threads it zeroes out the thread pointer, and also checks whether it's NULL before trying to stop it. Second, in function gfs2_remount_fs, it was returning if an error was logged by either of the two functions for gfs2_make_fs_ro and _rw, which caused it to bypass the online uevent at the bottom of the function. This removes that bypass in favor of just running the whole function, then returning the error. That way, unmounts and remounts won't hang forever.

---
 fs/gfs2/super.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
fs/gfs2/super.c +14 -7
index a6a325b2a78b..dc4ec1ca3d4e 100644
@@ -396,6 +396,7 @@ static int init_threads(struct gfs2_sbd *sdp)
 
 fail:
 	kthread_stop(sdp->sd_logd_process);
+	sdp->sd_logd_process = NULL;
 	return error;
 }
 
@@ -453,8 +454,12 @@ int gfs2_make_fs_rw(struct gfs2_sbd *sdp)
 	freeze_gh.gh_flags |= GL_NOCACHE;
 	gfs2_glock_dq_uninit(&freeze_gh);
 fail_threads:
-	kthread_stop(sdp->sd_quotad_process);
-	kthread_stop(sdp->sd_logd_process);
+	if (sdp->sd_quotad_process)
+		kthread_stop(sdp->sd_quotad_process);
+	sdp->sd_quotad_process = NULL;
+	if (sdp->sd_logd_process)
+		kthread_stop(sdp->sd_logd_process);
+	sdp->sd_logd_process = NULL;
 	return error;
 }
 
@@ -855,8 +860,12 @@ static int gfs2_make_fs_ro(struct gfs2_sbd *sdp)
 		return error;
 
 	flush_workqueue(gfs2_delete_workqueue);
-	kthread_stop(sdp->sd_quotad_process);
-	kthread_stop(sdp->sd_logd_process);
+	if (sdp->sd_quotad_process)
+		kthread_stop(sdp->sd_quotad_process);
+	sdp->sd_quotad_process = NULL;
+	if (sdp->sd_logd_process)
+		kthread_stop(sdp->sd_logd_process);
+	sdp->sd_logd_process = NULL;
 
 	gfs2_quota_sync(sdp->sd_vfs, 0);
 	gfs2_statfs_sync(sdp->sd_vfs, 0);
@@ -1273,8 +1282,6 @@ static int gfs2_remount_fs(struct super_block *sb, int *flags, char *data)
 			error = gfs2_make_fs_ro(sdp);
 		else
 			error = gfs2_make_fs_rw(sdp);
-		if (error)
-			return error;
 	}
 
 	sdp->sd_args = args;
@@ -1300,7 +1307,7 @@ static int gfs2_remount_fs(struct super_block *sb, int *flags, char *data)
 	spin_unlock(&gt->gt_spin);
 
 	gfs2_online_uevent(sdp);
-	return 0;
+	return error;
 }
 
 /**
Signature · 1 line
2.20.1
#2 lore raw

PATCH v3 2/19 gfs2: eliminate tr_num_revoke_rm

In reply to Bob Peterson

For its journal processing, gfs2 kept track of the number of buffers added and removed on a per-transaction basis. These values are used to calculate space needed in the journal. But while these calculations make sense for the number of buffers, they make no sense for revokes. Revokes are managed in their own list, linked from the superblock. So it's entirely unnecessary to keep separate per-transaction counts for revokes added and removed. A single count will do the same job. Therefore, this patch combines the transaction revokes into a single count.

---
 fs/gfs2/incore.h | 1 -
 fs/gfs2/log.c    | 3 +--
 fs/gfs2/trans.c  | 6 +++---
 3 files changed, 4 insertions(+), 6 deletions(-)
fs/gfs2/incore.h +0 -1
index 78c8e761b321..c54800385298 100644
@@ -507,7 +507,6 @@ struct gfs2_trans {
 	unsigned int tr_num_buf_rm;
 	unsigned int tr_num_databuf_rm;
 	unsigned int tr_num_revoke;
-	unsigned int tr_num_revoke_rm;
 
 	struct list_head tr_list;
 	struct list_head tr_databuf;
fs/gfs2/log.c +1 -2
index a7febb4bd400..76da6f046379 100644
@@ -887,7 +887,6 @@ static void gfs2_merge_trans(struct gfs2_trans *old, struct gfs2_trans *new)
 	old->tr_num_buf_rm	+= new->tr_num_buf_rm;
 	old->tr_num_databuf_rm	+= new->tr_num_databuf_rm;
 	old->tr_num_revoke	+= new->tr_num_revoke;
-	old->tr_num_revoke_rm	+= new->tr_num_revoke_rm;
 
 	list_splice_tail_init(&new->tr_databuf, &old->tr_databuf);
 	list_splice_tail_init(&new->tr_buf, &old->tr_buf);
@@ -909,7 +908,7 @@ static void log_refund(struct gfs2_sbd *sdp, struct gfs2_trans *tr)
 		set_bit(TR_ATTACHED, &tr->tr_flags);
 	}
 
-	sdp->sd_log_commited_revoke += tr->tr_num_revoke - tr->tr_num_revoke_rm;
+	sdp->sd_log_commited_revoke += tr->tr_num_revoke;
 	reserved = calc_reserved(sdp);
 	maxres = sdp->sd_log_blks_reserved + tr->tr_reserved;
 	gfs2_assert_withdraw(sdp, maxres >= reserved);
fs/gfs2/trans.c +3 -3
index d328da7cde36..7ebc0f99deb1 100644
@@ -80,10 +80,10 @@ static void gfs2_print_trans(struct gfs2_sbd *sdp, const struct gfs2_trans *tr)
 	fs_warn(sdp, "blocks=%u revokes=%u reserved=%u touched=%u\n",
 		tr->tr_blocks, tr->tr_revokes, tr->tr_reserved,
 		test_bit(TR_TOUCHED, &tr->tr_flags));
-	fs_warn(sdp, "Buf %u/%u Databuf %u/%u Revoke %u/%u\n",
+	fs_warn(sdp, "Buf %u/%u Databuf %u/%u Revoke %u\n",
 		tr->tr_num_buf_new, tr->tr_num_buf_rm,
 		tr->tr_num_databuf_new, tr->tr_num_databuf_rm,
-		tr->tr_num_revoke, tr->tr_num_revoke_rm);
+		tr->tr_num_revoke);
 }
 
 void gfs2_trans_end(struct gfs2_sbd *sdp)
@@ -266,7 +266,7 @@ void gfs2_trans_remove_revoke(struct gfs2_sbd *sdp, u64 blkno, unsigned int len)
 			gfs2_assert_withdraw(sdp, sdp->sd_log_num_revoke);
 			sdp->sd_log_num_revoke--;
 			kmem_cache_free(gfs2_bufdata_cachep, bd);
-			tr->tr_num_revoke_rm++;
+			tr->tr_num_revoke--;
 			if (--n == 0)
 				break;
 		}
Signature · 1 line
2.20.1
#3 lore raw

PATCH v3 3/19 gfs2: log which portion of the journal is replayed

In reply to Bob Peterson

When a journal is replayed, gfs2 logs a message similar to:

jid=X: Replaying journal...

This patch adds the tail and block number so that the range of the replayed block is also printed. These values will match the values shown if the journal is dumped with gfs2_edit -p journalX. The resulting output looks something like this:

jid=1: Replaying journal...0x28b7 to 0x2beb

This will allow us to better debug file system corruption problems.

---
 fs/gfs2/recovery.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/gfs2/recovery.c +2 -1
index fa575d1676b9..16422fdef2b9 100644
@@ -514,7 +514,8 @@ void gfs2_recover_func(struct work_struct *work)
 		}
 
 		t_tlck = ktime_get();
-		fs_info(sdp, "jid=%u: Replaying journal...\n", jd->jd_jid);
+		fs_info(sdp, "jid=%u: Replaying journal...0x%x to 0x%x\n",
+			jd->jd_jid, head.lh_tail, head.lh_blkno);
 
 		for (pass = 0; pass < 2; pass++) {
 			lops_before_scan(jd, &head, pass);
Signature · 1 line
2.20.1
#4 lore raw

PATCH v3 4/19 gfs2: Warn when a journal replay overwrites a rgrp with buffers

In reply to Bob Peterson

This patch adds some instrumentation in gfs2's journal replay that indicates when we're about to overwrite a rgrp for which we already have a valid buffer_head.

---
 fs/gfs2/lops.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
fs/gfs2/lops.c +20 -2
index 6af6a3cea967..2e8c6d02e112 100644
@@ -564,9 +564,27 @@ static int buf_lo_scan_elements(struct gfs2_jdesc *jd, u32 start,
 
 		if (gfs2_meta_check(sdp, bh_ip))
 			error = -EIO;
-		else
+		else {
+			struct gfs2_meta_header *mh =
+				(struct gfs2_meta_header *)bh_ip->b_data;
+
+			if (mh->mh_type == cpu_to_be32(GFS2_METATYPE_RG)) {
+				struct gfs2_rgrpd *rgd;
+
+				rgd = gfs2_blk2rgrpd(sdp, blkno, false);
+				if (rgd && rgd->rd_addr == blkno &&
+				    rgd->rd_bits && rgd->rd_bits->bi_bh) {
+					fs_info(sdp, "Replaying 0x%llx but we "
+						"already have a bh!\n",
+						(unsigned long long)blkno);
+					fs_info(sdp, "busy:%d, pinned:%d\n",
+						buffer_busy(rgd->rd_bits->bi_bh) ? 1 : 0,
+						buffer_pinned(rgd->rd_bits->bi_bh));
+					gfs2_dump_glock(NULL, rgd->rd_gl);
+				}
+			}
 			mark_buffer_dirty(bh_ip);
-
+		}
 		brelse(bh_log);
 		brelse(bh_ip);
 
Signature · 1 line
2.20.1
#5 lore raw

PATCH v3 4/19 gfs2: Warn when a journal replay overwrites a rgrp with buffers

Bob,

Quoted 3 lines · Bob Peterson

This patch adds some instrumentation in gfs2's journal replay that indicates when we're about to overwrite a rgrp for which we already have a valid buffer_head.

looks okay, but can you explain in the commit message when this problem will trigger and why that's a problem?

Thanks,
Andreas

Quoted 42 lines
---
 fs/gfs2/lops.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
fs/gfs2/lops.c +20 -3
index 6af6a3cea967..2e8c6d02e112 100644
@@ -564,9 +564,27 @@ static int buf_lo_scan_elements(struct gfs2_jdesc *jd, u32 start,
 
                if (gfs2_meta_check(sdp, bh_ip))
                        error = -EIO;
-               else
+               else {
+                       struct gfs2_meta_header *mh =
+                               (struct gfs2_meta_header *)bh_ip->b_data;
+
+                       if (mh->mh_type == cpu_to_be32(GFS2_METATYPE_RG)) {
+                               struct gfs2_rgrpd *rgd;
+
+                               rgd = gfs2_blk2rgrpd(sdp, blkno, false);
+                               if (rgd && rgd->rd_addr == blkno &&
+                                   rgd->rd_bits && rgd->rd_bits->bi_bh) {
+                                       fs_info(sdp, "Replaying 0x%llx but we "
+                                               "already have a bh!\n",
+                                               (unsigned long long)blkno);
+                                       fs_info(sdp, "busy:%d, pinned:%d\n",
+                                               buffer_busy(rgd->rd_bits->bi_bh) ? 1 : 0,
+                                               buffer_pinned(rgd->rd_bits->bi_bh));
+                                       gfs2_dump_glock(NULL, rgd->rd_gl);
+                               }
+                       }
                        mark_buffer_dirty(bh_ip);
-
+               }
                brelse(bh_log);
                brelse(bh_ip);
 
--

2.20.1

#6 lore raw

PATCH v3 5/19 gfs2: Introduce concept of a pending withdraw

In reply to Bob Peterson

File system withdraws can be delayed when inconsistencies are discovered when we cannot withdraw immediately, for example, when critical spin_locks are held. But delaying the withdraw can cause gfs2 to ignore the error and keep running for a short period of time. For example, an rgrp glock may be dequeued and demoted while there are still buffers that haven't been properly revoked, due to io errors writing to the journal.

This patch introduces a new concept of a pending withdraw, which means an inconsistency has been discovered and we need to withdraw at the earliest possible opportunity. In these cases, we aren't quite withdrawn yet, but we still need to not dequeue glocks and other critical things. If we dequeue the glocks and the withdraw results in our journal being replayed, the replay could overwrite data that's been modified by a different node that acquired the glock in the meantime.

---
 fs/gfs2/aops.c       |  4 ++--
 fs/gfs2/file.c       |  2 +-
 fs/gfs2/glock.c      |  7 +++----
 fs/gfs2/glops.c      |  2 +-
 fs/gfs2/incore.h     |  1 +
 fs/gfs2/log.c        | 20 ++++++++------------
 fs/gfs2/meta_io.c    |  6 +++---
 fs/gfs2/ops_fstype.c |  3 +--
 fs/gfs2/quota.c      |  2 +-
 fs/gfs2/super.c      |  6 +++---
 fs/gfs2/sys.c        |  2 +-
 fs/gfs2/util.h       | 12 ++++++++++++
 12 files changed, 37 insertions(+), 30 deletions(-)
fs/gfs2/aops.c +2 -2
index 05dd78f4b2b3..2bacce032a34 100644
@@ -521,7 +521,7 @@ static int __gfs2_readpage(void *file, struct page *page)
 		error = mpage_readpage(page, gfs2_block_map);
 	}
 
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+	if (unlikely(gfs2_withdrawn(sdp)))
 		return -EIO;
 
 	return error;
@@ -638,7 +638,7 @@ static int gfs2_readpages(struct file *file, struct address_space *mapping,
 	gfs2_glock_dq(&gh);
 out_uninit:
 	gfs2_holder_uninit(&gh);
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+	if (unlikely(gfs2_withdrawn(sdp)))
 		ret = -EIO;
 	return ret;
 }
fs/gfs2/file.c +1 -1
index 58a768e59712..bba39c73a43c 100644
@@ -1169,7 +1169,7 @@ static int gfs2_lock(struct file *file, int cmd, struct file_lock *fl)
 		cmd = F_SETLK;
 		fl->fl_type = F_UNLCK;
 	}
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags))) {
+	if (unlikely(gfs2_withdrawn(sdp))) {
 		if (fl->fl_type == F_UNLCK)
 			locks_lock_file_wait(file, fl);
 		return -EIO;
fs/gfs2/glock.c +3 -4
index 15c605cfcfc8..fe9f9a22426e 100644
@@ -547,7 +547,7 @@ __acquires(&gl->gl_lockref.lock)
 	unsigned int lck_flags = (unsigned int)(gh ? gh->gh_flags : 0);
 	int ret;
 
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) &&
+	if (unlikely(gfs2_withdrawn(sdp)) &&
 	    target != LM_ST_UNLOCKED)
 		return;
 	lck_flags &= (LM_FLAG_TRY | LM_FLAG_TRY_1CB | LM_FLAG_NOEXP |
@@ -584,8 +584,7 @@ __acquires(&gl->gl_lockref.lock)
 		}
 		else if (ret) {
 			fs_err(sdp, "lm_lock ret %d\n", ret);
-			GLOCK_BUG_ON(gl, !test_bit(SDF_SHUTDOWN,
-						   &sdp->sd_flags));
+			GLOCK_BUG_ON(gl, !gfs2_withdrawn(sdp));
 		}
 	} else { /* lock_nolock */
 		finish_xmote(gl, target);
@@ -1097,7 +1096,7 @@ int gfs2_glock_nq(struct gfs2_holder *gh)
 	struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
 	int error = 0;
 
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+	if (unlikely(gfs2_withdrawn(sdp)))
 		return -EIO;
 
 	if (test_bit(GLF_LRU, &gl->gl_flags))
fs/gfs2/glops.c +1 -1
index 78510ab91835..b9f36a85a4d4 100644
@@ -538,7 +538,7 @@ static int freeze_go_xmote_bh(struct gfs2_glock *gl, struct gfs2_holder *gh)
 			gfs2_consist(sdp);
 
 		/*  Initialize some head of the log stuff  */
-		if (!test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+		if (!gfs2_withdrawn(sdp)) {
 			sdp->sd_log_sequence = head.lh_sequence + 1;
 			gfs2_log_pointers_init(sdp, head.lh_blkno);
 		}
fs/gfs2/incore.h +1 -0
index c54800385298..003d9da937b4 100644
@@ -620,6 +620,7 @@ enum {
 	SDF_SKIP_DLM_UNLOCK	= 8,
 	SDF_FORCE_AIL_FLUSH     = 9,
 	SDF_AIL1_IO_ERROR	= 10,
+	SDF_WITHDRAW		= 11, /* Will withdraw eventually */
 };
 
 enum gfs2_freeze_state {
fs/gfs2/log.c +8 -12
index 76da6f046379..944aaf3d1816 100644
@@ -92,8 +92,7 @@ static void gfs2_remove_from_ail(struct gfs2_bufdata *bd)
 
 static int gfs2_ail1_start_one(struct gfs2_sbd *sdp,
 			       struct writeback_control *wbc,
-			       struct gfs2_trans *tr,
-			       bool *withdraw)
+			       struct gfs2_trans *tr)
 __releases(&sdp->sd_ail_lock)
 __acquires(&sdp->sd_ail_lock)
 {
@@ -112,7 +111,7 @@ __acquires(&sdp->sd_ail_lock)
 			    !test_and_set_bit(SDF_AIL1_IO_ERROR,
 					      &sdp->sd_flags)) {
 				gfs2_io_error_bh(sdp, bh);
-				*withdraw = true;
+				set_bit(SDF_WITHDRAW, &sdp->sd_flags);
 			}
 			list_move(&bd->bd_ail_st_list, &tr->tr_ail2_list);
 			continue;
@@ -153,7 +152,6 @@ void gfs2_ail1_flush(struct gfs2_sbd *sdp, struct writeback_control *wbc)
 	struct list_head *head = &sdp->sd_ail1_list;
 	struct gfs2_trans *tr;
 	struct blk_plug plug;
-	bool withdraw = false;
 
 	trace_gfs2_ail_flush(sdp, wbc, 1);
 	blk_start_plug(&plug);
@@ -162,12 +160,12 @@ void gfs2_ail1_flush(struct gfs2_sbd *sdp, struct writeback_control *wbc)
 	list_for_each_entry_reverse(tr, head, tr_list) {
 		if (wbc->nr_to_write <= 0)
 			break;
-		if (gfs2_ail1_start_one(sdp, wbc, tr, &withdraw))
+		if (gfs2_ail1_start_one(sdp, wbc, tr))
 			goto restart;
 	}
 	spin_unlock(&sdp->sd_ail_lock);
 	blk_finish_plug(&plug);
-	if (withdraw)
+	if (test_bit(SDF_WITHDRAW, &sdp->sd_flags))
 		gfs2_lm_withdraw(sdp, NULL);
 	trace_gfs2_ail_flush(sdp, wbc, 0);
 }
@@ -196,8 +194,7 @@ static void gfs2_ail1_start(struct gfs2_sbd *sdp)
  *
  */
 
-static void gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr,
-				bool *withdraw)
+static void gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr)
 {
 	struct gfs2_bufdata *bd, *s;
 	struct buffer_head *bh;
@@ -211,7 +208,7 @@ static void gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr,
 		if (!buffer_uptodate(bh) &&
 		    !test_and_set_bit(SDF_AIL1_IO_ERROR, &sdp->sd_flags)) {
 			gfs2_io_error_bh(sdp, bh);
-			*withdraw = true;
+			set_bit(SDF_WITHDRAW, &sdp->sd_flags);
 		}
 		list_move(&bd->bd_ail_st_list, &tr->tr_ail2_list);
 	}
@@ -229,11 +226,10 @@ static int gfs2_ail1_empty(struct gfs2_sbd *sdp)
 	struct gfs2_trans *tr, *s;
 	int oldest_tr = 1;
 	int ret;
-	bool withdraw = false;
 
 	spin_lock(&sdp->sd_ail_lock);
 	list_for_each_entry_safe_reverse(tr, s, &sdp->sd_ail1_list, tr_list) {
-		gfs2_ail1_empty_one(sdp, tr, &withdraw);
+		gfs2_ail1_empty_one(sdp, tr);
 		if (list_empty(&tr->tr_ail1_list) && oldest_tr)
 			list_move(&tr->tr_list, &sdp->sd_ail2_list);
 		else
@@ -242,7 +238,7 @@ static int gfs2_ail1_empty(struct gfs2_sbd *sdp)
 	ret = list_empty(&sdp->sd_ail1_list);
 	spin_unlock(&sdp->sd_ail_lock);
 
-	if (withdraw)
+	if (test_bit(SDF_WITHDRAW, &sdp->sd_flags))
 		gfs2_lm_withdraw(sdp, "fatal: I/O error(s)\n");
 
 	return ret;
fs/gfs2/meta_io.c +3 -3
index 3201342404a7..b125008f064f 100644
@@ -255,7 +255,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
 	struct buffer_head *bh, *bhs[2];
 	int num = 0;
 
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags))) {
+	if (unlikely(gfs2_withdrawn(sdp))) {
 		*bhp = NULL;
 		return -EIO;
 	}
@@ -313,7 +313,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
 
 int gfs2_meta_wait(struct gfs2_sbd *sdp, struct buffer_head *bh)
 {
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+	if (unlikely(gfs2_withdrawn(sdp)))
 		return -EIO;
 
 	wait_on_buffer(bh);
@@ -324,7 +324,7 @@ int gfs2_meta_wait(struct gfs2_sbd *sdp, struct buffer_head *bh)
 			gfs2_io_error_bh_wd(sdp, bh);
 		return -EIO;
 	}
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+	if (unlikely(gfs2_withdrawn(sdp)))
 		return -EIO;
 
 	return 0;
fs/gfs2/ops_fstype.c +1 -2
index abfaecde0e3d..075f64075ff6 100644
@@ -998,8 +998,7 @@ static int gfs2_lm_mount(struct gfs2_sbd *sdp, int silent)
 void gfs2_lm_unmount(struct gfs2_sbd *sdp)
 {
 	const struct lm_lockops *lm = sdp->sd_lockstruct.ls_ops;
-	if (likely(!test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) &&
-	    lm->lm_unmount)
+	if (likely(!gfs2_withdrawn(sdp)) && lm->lm_unmount)
 		lm->lm_unmount(sdp);
 }
 
fs/gfs2/quota.c +1 -1
index 2ae5a109eea7..a8dfc86fd682 100644
@@ -1478,7 +1478,7 @@ static void quotad_error(struct gfs2_sbd *sdp, const char *msg, int error)
 {
 	if (error == 0 || error == -EROFS)
 		return;
-	if (!test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+	if (!gfs2_withdrawn(sdp)) {
 		fs_err(sdp, "gfs2_quotad: %s error %d\n", msg, error);
 		sdp->sd_log_error = error;
 		wake_up(&sdp->sd_logd_waitq);
fs/gfs2/super.c +3 -3
index dc4ec1ca3d4e..378519beb6c3 100644
@@ -807,7 +807,7 @@ static void gfs2_dirty_inode(struct inode *inode, int flags)
 
 	if (!(flags & I_DIRTY_INODE))
 		return;
-	if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+	if (unlikely(gfs2_withdrawn(sdp)))
 		return;
 	if (!gfs2_glock_is_locked_by_me(ip->i_gl)) {
 		ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_EXCLUSIVE, 0, &gh);
@@ -856,7 +856,7 @@ static int gfs2_make_fs_ro(struct gfs2_sbd *sdp)
 
 	error = gfs2_glock_nq_init(sdp->sd_freeze_gl, LM_ST_SHARED, GL_NOCACHE,
 				   &freeze_gh);
-	if (error && !test_bit(SDF_SHUTDOWN, &sdp->sd_flags))
+	if (error && !gfs2_withdrawn(sdp))
 		return error;
 
 	flush_workqueue(gfs2_delete_workqueue);
@@ -1014,7 +1014,7 @@ static int gfs2_freeze(struct super_block *sb)
 	if (atomic_read(&sdp->sd_freeze_state) != SFS_UNFROZEN)
 		goto out;
 
-	if (test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+	if (gfs2_withdrawn(sdp)) {
 		error = -EINVAL;
 		goto out;
 	}
fs/gfs2/sys.c +1 -1
index 1787d295834e..b772ff5509bb 100644
@@ -121,7 +121,7 @@ static ssize_t freeze_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
 
 static ssize_t withdraw_show(struct gfs2_sbd *sdp, char *buf)
 {
-	unsigned int b = test_bit(SDF_SHUTDOWN, &sdp->sd_flags);
+	unsigned int b = gfs2_withdrawn(sdp);
 	return snprintf(buf, PAGE_SIZE, "%u\n", b);
 }
 
fs/gfs2/util.h +12 -0
index 9278fecba632..0df65555545f 100644
@@ -167,6 +167,18 @@ static inline unsigned int gfs2_tune_get_i(struct gfs2_tune *gt,
 	return x;
 }
 
+/**
+ * gfs2_withdrawn - test whether the file system is withdrawing or withdrawn
+ * @sdp: the superblock
+ */
+static inline bool gfs2_withdrawn(struct gfs2_sbd *sdp)
+{
+	if (test_bit(SDF_SHUTDOWN, &sdp->sd_flags) ||
+	    test_bit(SDF_WITHDRAW, &sdp->sd_flags))
+		return true;
+	return false;
+}
+
 #define gfs2_tune_get(sdp, field) \
 gfs2_tune_get_i(&(sdp)->sd_tune, &(sdp)->sd_tune.field)
 
Signature · 1 line
2.20.1
#7 lore raw

PATCH v3 5/19 gfs2: Introduce concept of a pending withdraw

Bob,

Quoted 8 lines · Bob Peterson

File system withdraws can be delayed when inconsistencies are discovered when we cannot withdraw immediately, for example, when critical spin_locks are held. But delaying the withdraw can cause gfs2 to ignore the error and keep running for a short period of time. For example, an rgrp glock may be dequeued and demoted while there are still buffers that haven't been properly revoked, due to io errors writing to the journal.

Quoted 8 lines

This patch introduces a new concept of a pending withdraw, which means an inconsistency has been discovered and we need to withdraw at the earliest possible opportunity. In these cases, we aren't quite withdrawn yet, but we still need to not dequeue glocks and other critical things. If we dequeue the glocks and the withdraw results in our journal being replayed, the replay could overwrite data that's been modified by a different node that acquired the glock in the meantime.

this is looking good. Maybe we can improve our terminology a bit
though: how about SDF_FAULTY instead of SDF_WITHDRAW? Also, SDF_SHUTDOWN really indicates that a filesystem is being or has been withdrawn, so maybe that should really be called SDF_WITHDRAW?

Thanks,
Andreas

Quoted 323 lines
---
 fs/gfs2/aops.c       |  4 ++--
 fs/gfs2/file.c       |  2 +-
 fs/gfs2/glock.c      |  7 +++----
 fs/gfs2/glops.c      |  2 +-
 fs/gfs2/incore.h     |  1 +
 fs/gfs2/log.c        | 20 ++++++++------------
 fs/gfs2/meta_io.c    |  6 +++---
 fs/gfs2/ops_fstype.c |  3 +--
 fs/gfs2/quota.c      |  2 +-
 fs/gfs2/super.c      |  6 +++---
 fs/gfs2/sys.c        |  2 +-
 fs/gfs2/util.h       | 12 ++++++++++++
 12 files changed, 37 insertions(+), 30 deletions(-)
fs/gfs2/aops.c +2 -2
index 05dd78f4b2b3..2bacce032a34 100644
@@ -521,7 +521,7 @@ static int __gfs2_readpage(void *file, struct page *page)
                error = mpage_readpage(page, gfs2_block_map);
        }
 
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+       if (unlikely(gfs2_withdrawn(sdp)))
                return -EIO;
 
        return error;
@@ -638,7 +638,7 @@ static int gfs2_readpages(struct file *file, struct address_space *mapping,
        gfs2_glock_dq(&gh);
 out_uninit:
        gfs2_holder_uninit(&gh);
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+       if (unlikely(gfs2_withdrawn(sdp)))
                ret = -EIO;
        return ret;
 }
fs/gfs2/file.c +1 -1
index 58a768e59712..bba39c73a43c 100644
@@ -1169,7 +1169,7 @@ static int gfs2_lock(struct file *file, int cmd, struct file_lock *fl)
                cmd = F_SETLK;
                fl->fl_type = F_UNLCK;
        }
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags))) {
+       if (unlikely(gfs2_withdrawn(sdp))) {
                if (fl->fl_type == F_UNLCK)
                        locks_lock_file_wait(file, fl);
                return -EIO;
fs/gfs2/glock.c +3 -4
index 15c605cfcfc8..fe9f9a22426e 100644
@@ -547,7 +547,7 @@ __acquires(&gl->gl_lockref.lock)
        unsigned int lck_flags = (unsigned int)(gh ? gh->gh_flags : 0);
        int ret;
 
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) &&
+       if (unlikely(gfs2_withdrawn(sdp)) &&
            target != LM_ST_UNLOCKED)
                return;
        lck_flags &= (LM_FLAG_TRY | LM_FLAG_TRY_1CB | LM_FLAG_NOEXP |
@@ -584,8 +584,7 @@ __acquires(&gl->gl_lockref.lock)
                }
                else if (ret) {
                        fs_err(sdp, "lm_lock ret %d\n", ret);
-                       GLOCK_BUG_ON(gl, !test_bit(SDF_SHUTDOWN,
-                                                  &sdp->sd_flags));
+                       GLOCK_BUG_ON(gl, !gfs2_withdrawn(sdp));
                }
        } else { /* lock_nolock */
                finish_xmote(gl, target);
@@ -1097,7 +1096,7 @@ int gfs2_glock_nq(struct gfs2_holder *gh)
        struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
        int error = 0;
 
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+       if (unlikely(gfs2_withdrawn(sdp)))
                return -EIO;
 
        if (test_bit(GLF_LRU, &gl->gl_flags))
fs/gfs2/glops.c +1 -1
index 78510ab91835..b9f36a85a4d4 100644
@@ -538,7 +538,7 @@ static int freeze_go_xmote_bh(struct gfs2_glock *gl, struct gfs2_holder *gh)
                        gfs2_consist(sdp);
 
                /*  Initialize some head of the log stuff  */
-               if (!test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+               if (!gfs2_withdrawn(sdp)) {
                        sdp->sd_log_sequence = head.lh_sequence + 1;
                        gfs2_log_pointers_init(sdp, head.lh_blkno);
                }
fs/gfs2/incore.h +1 -0
index c54800385298..003d9da937b4 100644
@@ -620,6 +620,7 @@ enum {
        SDF_SKIP_DLM_UNLOCK     = 8,
        SDF_FORCE_AIL_FLUSH     = 9,
        SDF_AIL1_IO_ERROR       = 10,
+       SDF_WITHDRAW            = 11, /* Will withdraw eventually */
 };
 
 enum gfs2_freeze_state {
fs/gfs2/log.c +8 -12
index 76da6f046379..944aaf3d1816 100644
@@ -92,8 +92,7 @@ static void gfs2_remove_from_ail(struct gfs2_bufdata *bd)
 
 static int gfs2_ail1_start_one(struct gfs2_sbd *sdp,
                               struct writeback_control *wbc,
-                              struct gfs2_trans *tr,
-                              bool *withdraw)
+                              struct gfs2_trans *tr)
 __releases(&sdp->sd_ail_lock)
 __acquires(&sdp->sd_ail_lock)
 {
@@ -112,7 +111,7 @@ __acquires(&sdp->sd_ail_lock)
                            !test_and_set_bit(SDF_AIL1_IO_ERROR,
                                              &sdp->sd_flags)) {
                                gfs2_io_error_bh(sdp, bh);
-                               *withdraw = true;
+                               set_bit(SDF_WITHDRAW, &sdp->sd_flags);
                        }
                        list_move(&bd->bd_ail_st_list, &tr->tr_ail2_list);
                        continue;
@@ -153,7 +152,6 @@ void gfs2_ail1_flush(struct gfs2_sbd *sdp, struct writeback_control *wbc)
        struct list_head *head = &sdp->sd_ail1_list;
        struct gfs2_trans *tr;
        struct blk_plug plug;
-       bool withdraw = false;
 
        trace_gfs2_ail_flush(sdp, wbc, 1);
        blk_start_plug(&plug);
@@ -162,12 +160,12 @@ void gfs2_ail1_flush(struct gfs2_sbd *sdp, struct writeback_control *wbc)
        list_for_each_entry_reverse(tr, head, tr_list) {
                if (wbc->nr_to_write <= 0)
                        break;
-               if (gfs2_ail1_start_one(sdp, wbc, tr, &withdraw))
+               if (gfs2_ail1_start_one(sdp, wbc, tr))
                        goto restart;
        }
        spin_unlock(&sdp->sd_ail_lock);
        blk_finish_plug(&plug);
-       if (withdraw)
+       if (test_bit(SDF_WITHDRAW, &sdp->sd_flags))
                gfs2_lm_withdraw(sdp, NULL);
        trace_gfs2_ail_flush(sdp, wbc, 0);
 }
@@ -196,8 +194,7 @@ static void gfs2_ail1_start(struct gfs2_sbd *sdp)
  *
  */
 
-static void gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr,
-                               bool *withdraw)
+static void gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr)
 {
        struct gfs2_bufdata *bd, *s;
        struct buffer_head *bh;
@@ -211,7 +208,7 @@ static void gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr,
                if (!buffer_uptodate(bh) &&
                    !test_and_set_bit(SDF_AIL1_IO_ERROR, &sdp->sd_flags)) {
                        gfs2_io_error_bh(sdp, bh);
-                       *withdraw = true;
+                       set_bit(SDF_WITHDRAW, &sdp->sd_flags);
                }
                list_move(&bd->bd_ail_st_list, &tr->tr_ail2_list);
        }
@@ -229,11 +226,10 @@ static int gfs2_ail1_empty(struct gfs2_sbd *sdp)
        struct gfs2_trans *tr, *s;
        int oldest_tr = 1;
        int ret;
-       bool withdraw = false;
 
        spin_lock(&sdp->sd_ail_lock);
        list_for_each_entry_safe_reverse(tr, s, &sdp->sd_ail1_list, tr_list) {
-               gfs2_ail1_empty_one(sdp, tr, &withdraw);
+               gfs2_ail1_empty_one(sdp, tr);
                if (list_empty(&tr->tr_ail1_list) && oldest_tr)
                        list_move(&tr->tr_list, &sdp->sd_ail2_list);
                else
@@ -242,7 +238,7 @@ static int gfs2_ail1_empty(struct gfs2_sbd *sdp)
        ret = list_empty(&sdp->sd_ail1_list);
        spin_unlock(&sdp->sd_ail_lock);
 
-       if (withdraw)
+       if (test_bit(SDF_WITHDRAW, &sdp->sd_flags))
                gfs2_lm_withdraw(sdp, "fatal: I/O error(s)\n");
 
        return ret;
fs/gfs2/meta_io.c +3 -3
index 3201342404a7..b125008f064f 100644
@@ -255,7 +255,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
        struct buffer_head *bh, *bhs[2];
        int num = 0;
 
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags))) {
+       if (unlikely(gfs2_withdrawn(sdp))) {
                *bhp = NULL;
                return -EIO;
        }
@@ -313,7 +313,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
 
 int gfs2_meta_wait(struct gfs2_sbd *sdp, struct buffer_head *bh)
 {
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+       if (unlikely(gfs2_withdrawn(sdp)))
                return -EIO;
 
        wait_on_buffer(bh);
@@ -324,7 +324,7 @@ int gfs2_meta_wait(struct gfs2_sbd *sdp, struct buffer_head *bh)
                        gfs2_io_error_bh_wd(sdp, bh);
                return -EIO;
        }
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+       if (unlikely(gfs2_withdrawn(sdp)))
                return -EIO;
 
        return 0;
fs/gfs2/ops_fstype.c +1 -2
index abfaecde0e3d..075f64075ff6 100644
@@ -998,8 +998,7 @@ static int gfs2_lm_mount(struct gfs2_sbd *sdp, int silent)
 void gfs2_lm_unmount(struct gfs2_sbd *sdp)
 {
        const struct lm_lockops *lm = sdp->sd_lockstruct.ls_ops;
-       if (likely(!test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) &&
-           lm->lm_unmount)
+       if (likely(!gfs2_withdrawn(sdp)) && lm->lm_unmount)
                lm->lm_unmount(sdp);
 }
 
fs/gfs2/quota.c +1 -1
index 2ae5a109eea7..a8dfc86fd682 100644
@@ -1478,7 +1478,7 @@ static void quotad_error(struct gfs2_sbd *sdp, const char *msg, int error)
 {
        if (error == 0 || error == -EROFS)
                return;
-       if (!test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+       if (!gfs2_withdrawn(sdp)) {
                fs_err(sdp, "gfs2_quotad: %s error %d\n", msg, error);
                sdp->sd_log_error = error;
                wake_up(&sdp->sd_logd_waitq);
fs/gfs2/super.c +3 -3
index dc4ec1ca3d4e..378519beb6c3 100644
@@ -807,7 +807,7 @@ static void gfs2_dirty_inode(struct inode *inode, int flags)
 
        if (!(flags & I_DIRTY_INODE))
                return;
-       if (unlikely(test_bit(SDF_SHUTDOWN, &sdp->sd_flags)))
+       if (unlikely(gfs2_withdrawn(sdp)))
                return;
        if (!gfs2_glock_is_locked_by_me(ip->i_gl)) {
                ret = gfs2_glock_nq_init(ip->i_gl, LM_ST_EXCLUSIVE, 0, &gh);
@@ -856,7 +856,7 @@ static int gfs2_make_fs_ro(struct gfs2_sbd *sdp)
 
        error = gfs2_glock_nq_init(sdp->sd_freeze_gl, LM_ST_SHARED, GL_NOCACHE,
                                   &freeze_gh);
-       if (error && !test_bit(SDF_SHUTDOWN, &sdp->sd_flags))
+       if (error && !gfs2_withdrawn(sdp))
                return error;
 
        flush_workqueue(gfs2_delete_workqueue);
@@ -1014,7 +1014,7 @@ static int gfs2_freeze(struct super_block *sb)
        if (atomic_read(&sdp->sd_freeze_state) != SFS_UNFROZEN)
                goto out;
 
-       if (test_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+       if (gfs2_withdrawn(sdp)) {
                error = -EINVAL;
                goto out;
        }
fs/gfs2/sys.c +1 -1
index 1787d295834e..b772ff5509bb 100644
@@ -121,7 +121,7 @@ static ssize_t freeze_store(struct gfs2_sbd *sdp, const char *buf, size_t len)
 
 static ssize_t withdraw_show(struct gfs2_sbd *sdp, char *buf)
 {
-       unsigned int b = test_bit(SDF_SHUTDOWN, &sdp->sd_flags);
+       unsigned int b = gfs2_withdrawn(sdp);
        return snprintf(buf, PAGE_SIZE, "%u\n", b);
 }
 
fs/gfs2/util.h +12 -1
index 9278fecba632..0df65555545f 100644
@@ -167,6 +167,18 @@ static inline unsigned int gfs2_tune_get_i(struct gfs2_tune *gt,
        return x;
 }
 
+/**
+ * gfs2_withdrawn - test whether the file system is withdrawing or withdrawn
+ * @sdp: the superblock
+ */
+static inline bool gfs2_withdrawn(struct gfs2_sbd *sdp)
+{
+       if (test_bit(SDF_SHUTDOWN, &sdp->sd_flags) ||
+           test_bit(SDF_WITHDRAW, &sdp->sd_flags))
+               return true;
+       return false;
+}
+
 #define gfs2_tune_get(sdp, field) \
 gfs2_tune_get_i(&(sdp)->sd_tune, &(sdp)->sd_tune.field)
 
--

2.20.1

#8 lore raw

PATCH v3 6/19 gfs2: log error reform

In reply to Bob Peterson

Before this patch, gfs2 kept track of journal io errors in two places sd_log_error and the SDF_AIL1_IO_ERROR flag in sd_flags. This patch consolidates the two into sd_log_error so that it reflects the first error encountered writing to the journal. In future patches, we will take advantage of this by checking this value rather than having to check both when reacting to io errors.

In addition, this fixes a tight loop in unmount: If buffers get on the ail1 list and an io error occurs elsewhere, the ail1 list would never be cleared because they were always busy. So unmount would hang, waiting for the ail1 list to empty.

---
 fs/gfs2/incore.h |  5 ++---
 fs/gfs2/log.c    | 20 +++++++++++++++-----
 fs/gfs2/quota.c  |  2 +-
 3 files changed, 18 insertions(+), 9 deletions(-)
fs/gfs2/incore.h +2 -3
index 003d9da937b4..e16ab4c98072 100644
@@ -619,8 +619,7 @@ enum {
 	SDF_RORECOVERY		= 7, /* read only recovery */
 	SDF_SKIP_DLM_UNLOCK	= 8,
 	SDF_FORCE_AIL_FLUSH     = 9,
-	SDF_AIL1_IO_ERROR	= 10,
-	SDF_WITHDRAW		= 11, /* Will withdraw eventually */
+	SDF_WITHDRAW		= 10, /* Will withdraw eventually */
 };
 
 enum gfs2_freeze_state {
@@ -829,7 +828,7 @@ struct gfs2_sbd {
 	atomic_t sd_log_in_flight;
 	struct bio *sd_log_bio;
 	wait_queue_head_t sd_log_flush_wait;
-	int sd_log_error;
+	int sd_log_error; /* First log error */
 
 	atomic_t sd_reserving_log;
 	wait_queue_head_t sd_reserving_log_wait;
fs/gfs2/log.c +15 -5
index 944aaf3d1816..33ef2cb570e2 100644
@@ -108,8 +108,7 @@ __acquires(&sdp->sd_ail_lock)
 
 		if (!buffer_busy(bh)) {
 			if (!buffer_uptodate(bh) &&
-			    !test_and_set_bit(SDF_AIL1_IO_ERROR,
-					      &sdp->sd_flags)) {
+			    !cmpxchg(&sdp->sd_log_error, 0, -EIO)) {
 				gfs2_io_error_bh(sdp, bh);
 				set_bit(SDF_WITHDRAW, &sdp->sd_flags);
 			}
@@ -203,10 +202,21 @@ static void gfs2_ail1_empty_one(struct gfs2_sbd *sdp, struct gfs2_trans *tr)
 					 bd_ail_st_list) {
 		bh = bd->bd_bh;
 		gfs2_assert(sdp, bd->bd_tr == tr);
-		if (buffer_busy(bh))
+		/**
+		 * If another process flagged an io error, e.g. writing to the
+		 * journal, error all other bhs and move them off the ail1 to
+		 * prevent a tight loop when unmount tries to flush ail1,
+		 * regardless of whether they're still busy. If no outside
+		 * errors were found and the buffer is busy, move to the next.
+		 * If the ail buffer is not busy and caught an error, flag it
+		 * for others.
+		 */
+		if (sdp->sd_log_error) {
+			gfs2_io_error_bh(sdp, bh);
+		} else if (buffer_busy(bh)) {
 			continue;
-		if (!buffer_uptodate(bh) &&
-		    !test_and_set_bit(SDF_AIL1_IO_ERROR, &sdp->sd_flags)) {
+		} else if (!buffer_uptodate(bh) &&
+			   !cmpxchg(&sdp->sd_log_error, 0, -EIO)) {
 			gfs2_io_error_bh(sdp, bh);
 			set_bit(SDF_WITHDRAW, &sdp->sd_flags);
 		}
fs/gfs2/quota.c +1 -1
index a8dfc86fd682..8871fca9102f 100644
@@ -1480,7 +1480,7 @@ static void quotad_error(struct gfs2_sbd *sdp, const char *msg, int error)
 		return;
 	if (!gfs2_withdrawn(sdp)) {
 		fs_err(sdp, "gfs2_quotad: %s error %d\n", msg, error);
-		sdp->sd_log_error = error;
+		cmpxchg(&sdp->sd_log_error, 0, error);
 		wake_up(&sdp->sd_logd_waitq);
 	}
 }
Signature · 1 line
2.20.1
#9 lore raw

PATCH v3 7/19 gfs2: Only complain the first time an io error occurs in quota or log

In reply to Bob Peterson

Before this patch, all io errors received by the quota daemon or the logd daemon would cause a complaint message to be issued, such as:

gfs2: fsid=dm-13.0: Error 10 writing to journal, jid=0

This patch changes it so that the error message is only issued the first time the error is encountered.

Also, before this patch function gfs2_end_log_write did not set the sd_log_error value, so log errors would not cause the file system to be withdrawn. This patch sets the error code so the file system is properly withdrawn if an io error is encountered writing to the journal.

WARNING: This change in function breaks check xfstests generic/441 and causes it to fail: io errors writing to the log should cause a file system to be withdrawn, and no further operations are tolerated.

---
 fs/gfs2/lops.c  | 5 +++--
 fs/gfs2/quota.c | 4 ++--
 2 files changed, 5 insertions(+), 4 deletions(-)
fs/gfs2/lops.c +3 -2
index 2e8c6d02e112..065867b5f7c8 100644
@@ -211,8 +211,9 @@ static void gfs2_end_log_write(struct bio *bio)
 	struct bvec_iter_all iter_all;
 
 	if (bio->bi_status) {
-		fs_err(sdp, "Error %d writing to journal, jid=%u\n",
-		       bio->bi_status, sdp->sd_jdesc->jd_jid);
+		if (!cmpxchg(&sdp->sd_log_error, 0, bio->bi_status))
+			fs_err(sdp, "Error %d writing to journal, jid=%u\n",
+			       bio->bi_status, sdp->sd_jdesc->jd_jid);
 		wake_up(&sdp->sd_logd_waitq);
 	}
 
fs/gfs2/quota.c +2 -2
index 8871fca9102f..acb6cbdd1431 100644
@@ -1479,8 +1479,8 @@ static void quotad_error(struct gfs2_sbd *sdp, const char *msg, int error)
 	if (error == 0 || error == -EROFS)
 		return;
 	if (!gfs2_withdrawn(sdp)) {
-		fs_err(sdp, "gfs2_quotad: %s error %d\n", msg, error);
-		cmpxchg(&sdp->sd_log_error, 0, error);
+		if (!cmpxchg(&sdp->sd_log_error, 0, error))
+			fs_err(sdp, "gfs2_quotad: %s error %d\n", msg, error);
 		wake_up(&sdp->sd_logd_waitq);
 	}
 }
Signature · 1 line
2.20.1
#10 lore raw

PATCH v3 8/19 gfs2: Stop ail1 wait loop when withdrawn

In reply to Bob Peterson

Before this patch, function gfs2_log_flush could get into an infinite loop trying to clear out its ail1 list. If the file system was withdrawn (or pending withdraw) due to a problem with writing the ail1 list, it would never clear out the list, and therefore, would loop infinitely. This patch changes function gfs2_log_flush so that it does while (!gfs2_withdraw(sdp)) rather than while (;;).

---
 fs/gfs2/log.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/gfs2/log.c +1 -1
index 33ef2cb570e2..6169276aa9e6 100644
@@ -854,7 +854,7 @@ void gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl, u32 flags)
 
 	if (!(flags & GFS2_LOG_HEAD_FLUSH_NORMAL)) {
 		if (!sdp->sd_log_idle) {
-			for (;;) {
+			while (!gfs2_withdrawn(sdp)) {
 				gfs2_ail1_start(sdp);
 				gfs2_ail1_wait(sdp);
 				if (gfs2_ail1_empty(sdp))
Signature · 1 line
2.20.1
#11 lore raw

PATCH v3 9/19 gfs2: Ignore recovery attempts if gfs2 has io error or is withdrawn

In reply to Bob Peterson

This patch addresses various problems with gfs2/dlm recovery.

For example, suppose a node with a bunch of gfs2 mounts suddenly reboots due to kernel panic, and dlm determines it should perform recovery. DLM does so from a pseudo-state machine calling various callbacks into lock_dlm to perform a sequence of steps. It uses generation numbers and recover bits in dlm "control" lock lvbs.

Now suppose another node tries to recover the failed node's journal, but in so doing, encounters an IO error or withdraws due to unforeseen circumstances, such as an hba driver failure. In these cases, the recovery would eventually bail out, but it would still update its generation number in the lvb. The other nodes would all see the newer generation number and think they don't need to do recovery because the generation number is newer than the last one they saw, and therefore someone else has already taken care of it.

If the file system has an io error or is withdrawn, it cannot safely replay any journals (its own or others) but someone else still needs to do it. Therefore we don't want it messing with the journal recovery generation numbers: the local generation numbers eventually get put into the lvb generation numbers to be seen by all nodes.

This patch adds checks to many of the callbacks used by dlm in its recovery state machine so that the functions are ignored and skipped if an io error has occurred or if the file system was withdraw.

---
 fs/gfs2/lock_dlm.c | 18 ++++++++++++++++++
 fs/gfs2/util.c     | 15 +++++++--------
 2 files changed, 25 insertions(+), 8 deletions(-)
fs/gfs2/lock_dlm.c +18 -0
index 31df26ed7854..9329f86ffcbe 100644
@@ -1081,6 +1081,10 @@ static void gdlm_recover_prep(void *arg)
 	struct gfs2_sbd *sdp = arg;
 	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recover_prep ignored due to withdraw.\n");
+		return;
+	}
 	spin_lock(&ls->ls_recover_spin);
 	ls->ls_recover_block = ls->ls_recover_start;
 	set_bit(DFL_DLM_RECOVERY, &ls->ls_recover_flags);
@@ -1103,6 +1107,11 @@ static void gdlm_recover_slot(void *arg, struct dlm_slot *slot)
 	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
 	int jid = slot->slot - 1;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recover_slot jid %d ignored due to withdraw.\n",
+		       jid);
+		return;
+	}
 	spin_lock(&ls->ls_recover_spin);
 	if (ls->ls_recover_size < jid + 1) {
 		fs_err(sdp, "recover_slot jid %d gen %u short size %d\n",
@@ -1127,6 +1136,10 @@ static void gdlm_recover_done(void *arg, struct dlm_slot *slots, int num_slots,
 	struct gfs2_sbd *sdp = arg;
 	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recover_done ignored due to withdraw.\n");
+		return;
+	}
 	/* ensure the ls jid arrays are large enough */
 	set_recover_size(sdp, slots, num_slots);
 
@@ -1154,6 +1167,11 @@ static void gdlm_recovery_result(struct gfs2_sbd *sdp, unsigned int jid,
 {
 	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recovery_result jid %d ignored due to withdraw.\n",
+		       jid);
+		return;
+	}
 	if (test_bit(DFL_NO_DLM_OPS, &ls->ls_recover_flags))
 		return;
 
fs/gfs2/util.c +7 -8
index 0a814ccac41d..7eaea6dfe1cf 100644
@@ -259,14 +259,13 @@ void gfs2_io_error_bh_i(struct gfs2_sbd *sdp, struct buffer_head *bh,
 			const char *function, char *file, unsigned int line,
 			bool withdraw)
 {
-	if (!test_bit(SDF_SHUTDOWN, &sdp->sd_flags))
-		fs_err(sdp,
-		       "fatal: I/O error\n"
-		       "  block = %llu\n"
-		       "  function = %s, file = %s, line = %u\n",
-		       (unsigned long long)bh->b_blocknr,
-		       function, file, line);
+	if (gfs2_withdrawn(sdp))
+		return;
+
+	fs_err(sdp, "fatal: I/O error\n"
+	       "  block = %llu\n"
+	       "  function = %s, file = %s, line = %u\n",
+	       (unsigned long long)bh->b_blocknr, function, file, line);
 	if (withdraw)
 		gfs2_lm_withdraw(sdp, NULL);
 }
-
Signature · 1 line
2.20.1
#12 lore raw

PATCH v3 9/19 gfs2: Ignore recovery attempts if gfs2 has io error or is withdrawn

Hi,

Quoted 25 lines · Bob Peterson

This patch addresses various problems with gfs2/dlm recovery.

For example, suppose a node with a bunch of gfs2 mounts suddenly reboots due to kernel panic, and dlm determines it should perform recovery. DLM does so from a pseudo-state machine calling various callbacks into lock_dlm to perform a sequence of steps. It uses generation numbers and recover bits in dlm "control" lock lvbs.

Now suppose another node tries to recover the failed node's journal, but in so doing, encounters an IO error or withdraws due to unforeseen circumstances, such as an hba driver failure. In these cases, the recovery would eventually bail out, but it would still update its generation number in the lvb. The other nodes would all see the newer generation number and think they don't need to do recovery because the generation number is newer than the last one they saw, and therefore someone else has already taken care of it.

If the file system has an io error or is withdrawn, it cannot safely replay any journals (its own or others) but someone else still needs to do it. Therefore we don't want it messing with the journal recovery generation numbers: the local generation numbers eventually get put into the lvb generation numbers to be seen by all nodes.

Quoted 6 lines

This patch adds checks to many of the callbacks used by dlm in its recovery state machine so that the functions are ignored and skipped if an io error has occurred or if the file system was withdraw.

These should probably propagate the error back to the caller of the recovery request. We do have a proper notification system for failed recovery via uevents,

Steve.

Quoted 81 lines
---
  fs/gfs2/lock_dlm.c | 18 ++++++++++++++++++
  fs/gfs2/util.c     | 15 +++++++--------
  2 files changed, 25 insertions(+), 8 deletions(-)
fs/gfs2/lock_dlm.c +18 -0
index 31df26ed7854..9329f86ffcbe 100644
@@ -1081,6 +1081,10 @@ static void gdlm_recover_prep(void *arg)
  	struct gfs2_sbd *sdp = arg;
  	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recover_prep ignored due to withdraw.\n");
+		return;
+	}
  	spin_lock(&ls->ls_recover_spin);
  	ls->ls_recover_block = ls->ls_recover_start;
  	set_bit(DFL_DLM_RECOVERY, &ls->ls_recover_flags);
@@ -1103,6 +1107,11 @@ static void gdlm_recover_slot(void *arg, struct dlm_slot *slot)
  	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
  	int jid = slot->slot - 1;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recover_slot jid %d ignored due to withdraw.\n",
+		       jid);
+		return;
+	}
  	spin_lock(&ls->ls_recover_spin);
  	if (ls->ls_recover_size < jid + 1) {
  		fs_err(sdp, "recover_slot jid %d gen %u short size %d\n",
@@ -1127,6 +1136,10 @@ static void gdlm_recover_done(void *arg, struct dlm_slot *slots, int num_slots,
  	struct gfs2_sbd *sdp = arg;
  	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recover_done ignored due to withdraw.\n");
+		return;
+	}
  	/* ensure the ls jid arrays are large enough */
  	set_recover_size(sdp, slots, num_slots);
 
@@ -1154,6 +1167,11 @@ static void gdlm_recovery_result(struct gfs2_sbd *sdp, unsigned int jid,
  {
  	struct lm_lockstruct *ls = &sdp->sd_lockstruct;
 
+	if (gfs2_withdrawn(sdp)) {
+		fs_err(sdp, "recovery_result jid %d ignored due to withdraw.\n",
+		       jid);
+		return;
+	}
  	if (test_bit(DFL_NO_DLM_OPS, &ls->ls_recover_flags))
  		return;
 
fs/gfs2/util.c +7 -8
index 0a814ccac41d..7eaea6dfe1cf 100644
@@ -259,14 +259,13 @@ void gfs2_io_error_bh_i(struct gfs2_sbd *sdp, struct buffer_head *bh,
  			const char *function, char *file, unsigned int line,
  			bool withdraw)
  {
-	if (!test_bit(SDF_SHUTDOWN, &sdp->sd_flags))
-		fs_err(sdp,
-		       "fatal: I/O error\n"
-		       "  block = %llu\n"
-		       "  function = %s, file = %s, line = %u\n",
-		       (unsigned long long)bh->b_blocknr,
-		       function, file, line);
+	if (gfs2_withdrawn(sdp))
+		return;
+
+	fs_err(sdp, "fatal: I/O error\n"
+	       "  block = %llu\n"
+	       "  function = %s, file = %s, line = %u\n",
+	       (unsigned long long)bh->b_blocknr, function, file, line);
  	if (withdraw)
  		gfs2_lm_withdraw(sdp, NULL);
  }
-
#13 lore raw

PATCH v3 10/19 gfs2: move check_journal_clean to util.c for future use

In reply to Bob Peterson

Before this patch function check_journal_clean was in ops_fstype.c. This patch moves it to util.c so we can make use of it elsewhere in a future patch.

---
 fs/gfs2/ops_fstype.c | 42 -----------------------------------------
 fs/gfs2/util.c       | 45 ++++++++++++++++++++++++++++++++++++++++++++
 fs/gfs2/util.h       |  1 +
 3 files changed, 46 insertions(+), 42 deletions(-)
fs/gfs2/ops_fstype.c +0 -42
index 075f64075ff6..8a09f3aa609c 100644
@@ -590,48 +590,6 @@ static int gfs2_jindex_hold(struct gfs2_sbd *sdp, struct gfs2_holder *ji_gh)
 	return error;
 }
 
-/**
- * check_journal_clean - Make sure a journal is clean for a spectator mount
- * @sdp: The GFS2 superblock
- * @jd: The journal descriptor
- *
- * Returns: 0 if the journal is clean or locked, else an error
- */
-static int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd)
-{
-	int error;
-	struct gfs2_holder j_gh;
-	struct gfs2_log_header_host head;
-	struct gfs2_inode *ip;
-
-	ip = GFS2_I(jd->jd_inode);
-	error = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, LM_FLAG_NOEXP |
-				   GL_EXACT | GL_NOCACHE, &j_gh);
-	if (error) {
-		fs_err(sdp, "Error locking journal for spectator mount.\n");
-		return -EPERM;
-	}
-	error = gfs2_jdesc_check(jd);
-	if (error) {
-		fs_err(sdp, "Error checking journal for spectator mount.\n");
-		goto out_unlock;
-	}
-	error = gfs2_find_jhead(jd, &head);
-	if (error) {
-		fs_err(sdp, "Error parsing journal for spectator mount.\n");
-		goto out_unlock;
-	}
-	if (!(head.lh_flags & GFS2_LOG_HEAD_UNMOUNT)) {
-		error = -EPERM;
-		fs_err(sdp, "jid=%u: Journal is dirty, so the first mounter "
-		       "must not be a spectator.\n", jd->jd_jid);
-	}
-
-out_unlock:
-	gfs2_glock_dq_uninit(&j_gh);
-	return error;
-}
-
 static int init_journal(struct gfs2_sbd *sdp, int undo)
 {
 	struct inode *master = d_inode(sdp->sd_master_dir);
fs/gfs2/util.c +45 -0

58 lines not on this page · raw message

fs/gfs2/util.h +1 -0
index 0df65555545f..49d46865dceb 100644
@@ -131,6 +131,7 @@ static inline void gfs2_metatype_set(struct buffer_head *bh, u16 type,
 
 int gfs2_io_error_i(struct gfs2_sbd *sdp, const char *function,
 		    char *file, unsigned int line);
+int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd);
 
 #define gfs2_io_error(sdp) \
 gfs2_io_error_i((sdp), __func__, __FILE__, __LINE__);
Signature · 1 line
2.20.1
#14 lore raw

PATCH v3 11/19 gfs2: Allow some glocks to be used during withdraw

In reply to Bob Peterson

Before this patch, when a file system was withdrawn, all further attempts to enqueue or promote glocks were rejected and returned -EIO. This is only important for media-backed glocks like inode and rgrp glocks. All other glocks may be safely used because there is no potential for metadata corruption. This patch allows some glocks to be used even after the file system is withdrawn. This is accomplished with a new glops flag, GLOF_JOURNALED, which tells us which inodes cannot be safely manipulated after withdraw. This facilitates future patches that enhance fs withdraw.

---
 fs/gfs2/glock.c      |  7 +++++--
 fs/gfs2/glops.c      | 16 +++++++++++++---
 fs/gfs2/glops.h      |  3 ++-
 fs/gfs2/incore.h     |  8 ++++++++
 fs/gfs2/inode.c      | 14 ++++++++++----
 fs/gfs2/ops_fstype.c | 14 +++++++++++---
 fs/gfs2/sys.c        | 12 ++++++++++--
 7 files changed, 59 insertions(+), 15 deletions(-)
fs/gfs2/glock.c +5 -2
index fe9f9a22426e..eeb84d8d21f3 100644
@@ -548,7 +548,7 @@ __acquires(&gl->gl_lockref.lock)
 	int ret;
 
 	if (unlikely(gfs2_withdrawn(sdp)) &&
-	    target != LM_ST_UNLOCKED)
+	    (glops->go_flags & GLOF_JOURNALED) && target != LM_ST_UNLOCKED)
 		return;
 	lck_flags &= (LM_FLAG_TRY | LM_FLAG_TRY_1CB | LM_FLAG_NOEXP |
 		      LM_FLAG_PRIORITY);
@@ -1096,7 +1096,8 @@ int gfs2_glock_nq(struct gfs2_holder *gh)
 	struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
 	int error = 0;
 
-	if (unlikely(gfs2_withdrawn(sdp)))
+	if (unlikely(gfs2_withdrawn(sdp)) &&
+	    (gl->gl_ops->go_flags & GLOF_JOURNALED))
 		return -EIO;
 
 	if (test_bit(GLF_LRU, &gl->gl_flags))
@@ -1760,6 +1761,8 @@ static const char *gflags2str(char *buf, const struct gfs2_glock *gl)
 		*p++ = 'o';
 	if (test_bit(GLF_BLOCKING, gflags))
 		*p++ = 'b';
+	if (gl->gl_ops->go_flags & GLOF_JOURNALED)
+		*p++ = 'j';
 	*p = 0;
 	return buf;
 }
fs/gfs2/glops.c +13 -3
index b9f36a85a4d4..d2b44bdca17e 100644
@@ -584,7 +584,7 @@ const struct gfs2_glock_operations gfs2_meta_glops = {
 	.go_type = LM_TYPE_META,
 };
 
-const struct gfs2_glock_operations gfs2_inode_glops = {
+const struct gfs2_glock_operations gfs2_sys_inode_glops = {
 	.go_sync = inode_go_sync,
 	.go_inval = inode_go_inval,
 	.go_demote_ok = inode_go_demote_ok,
@@ -594,6 +594,16 @@ const struct gfs2_glock_operations gfs2_inode_glops = {
 	.go_flags = GLOF_ASPACE | GLOF_LRU,
 };
 
+const struct gfs2_glock_operations gfs2_user_inode_glops = {
+	.go_sync = inode_go_sync,
+	.go_inval = inode_go_inval,
+	.go_demote_ok = inode_go_demote_ok,
+	.go_lock = inode_go_lock,
+	.go_dump = inode_go_dump,
+	.go_type = LM_TYPE_INODE,
+	.go_flags = GLOF_ASPACE | GLOF_LRU | GLOF_JOURNALED,
+};
+
 const struct gfs2_glock_operations gfs2_rgrp_glops = {
 	.go_sync = rgrp_go_sync,
 	.go_inval = rgrp_go_inval,
@@ -601,7 +611,7 @@ const struct gfs2_glock_operations gfs2_rgrp_glops = {
 	.go_unlock = gfs2_rgrp_go_unlock,
 	.go_dump = gfs2_rgrp_dump,
 	.go_type = LM_TYPE_RGRP,
-	.go_flags = GLOF_LVB,
+	.go_flags = GLOF_LVB | GLOF_JOURNALED,
 };
 
 const struct gfs2_glock_operations gfs2_freeze_glops = {
@@ -637,7 +647,7 @@ const struct gfs2_glock_operations gfs2_journal_glops = {
 
 const struct gfs2_glock_operations *gfs2_glops_list[] = {
 	[LM_TYPE_META] = &gfs2_meta_glops,
-	[LM_TYPE_INODE] = &gfs2_inode_glops,
+	[LM_TYPE_INODE] = &gfs2_user_inode_glops,
 	[LM_TYPE_RGRP] = &gfs2_rgrp_glops,
 	[LM_TYPE_IOPEN] = &gfs2_iopen_glops,
 	[LM_TYPE_FLOCK] = &gfs2_flock_glops,
fs/gfs2/glops.h +2 -1
index 8ed1857c1a8d..63130a5959e1 100644
@@ -15,7 +15,8 @@
 extern struct workqueue_struct *gfs2_freeze_wq;
 
 extern const struct gfs2_glock_operations gfs2_meta_glops;
-extern const struct gfs2_glock_operations gfs2_inode_glops;
+extern const struct gfs2_glock_operations gfs2_user_inode_glops;
+extern const struct gfs2_glock_operations gfs2_sys_inode_glops;
 extern const struct gfs2_glock_operations gfs2_rgrp_glops;
 extern const struct gfs2_glock_operations gfs2_freeze_glops;
 extern const struct gfs2_glock_operations gfs2_iopen_glops;
fs/gfs2/incore.h +8 -0
index e16ab4c98072..b16b82e5b85a 100644
@@ -250,6 +250,14 @@ struct gfs2_glock_operations {
 #define GLOF_ASPACE 1
 #define GLOF_LVB    2
 #define GLOF_LRU    4
+/**
+ * The GLOF_JOURNALED flag marks objects that are journaled, which means we
+ * need to treat them specially when a file system withdraw occurs. Inodes
+ * like the journal inode itself are okay to manipulate after withdraw, but
+ * user files need to be protected from journal replay after their glock has
+ * been granted to another node.
+ */
+#define GLOF_JOURNALED 8
 };
 
 enum {
fs/gfs2/inode.c +10 -4
index 998051c4aea7..2fefd8581965 100644
@@ -141,7 +141,8 @@ struct inode *gfs2_inode_lookup(struct super_block *sb, unsigned int type,
 		struct gfs2_sbd *sdp = GFS2_SB(inode);
 		ip->i_no_formal_ino = no_formal_ino;
 
-		error = gfs2_glock_get(sdp, no_addr, &gfs2_inode_glops, CREATE, &ip->i_gl);
+		error = gfs2_glock_get(sdp, no_addr, &gfs2_user_inode_glops,
+				       CREATE, &ip->i_gl);
 		if (unlikely(error))
 			goto fail;
 		flush_delayed_work(&ip->i_gl->gl_work);
@@ -248,6 +249,8 @@ struct inode *gfs2_lookup_simple(struct inode *dip, const char *name)
 {
 	struct qstr qstr;
 	struct inode *inode;
+	struct gfs2_inode *ip;
+
 	gfs2_str2qstr(&qstr, name);
 	inode = gfs2_lookupi(dip, &qstr, 1);
 	/* gfs2_lookupi has inconsistent callers: vfs
@@ -257,8 +260,10 @@ struct inode *gfs2_lookup_simple(struct inode *dip, const char *name)
 	 */
 	if (inode == NULL)
 		return ERR_PTR(-ENOENT);
-	else
-		return inode;
+
+	ip = GFS2_I(inode);
+	ip->i_gl->gl_ops = &gfs2_sys_inode_glops;
+	return inode;
 }
 
 
@@ -703,7 +708,8 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
 
 	gfs2_set_inode_blocks(inode, blocks);
 
-	error = gfs2_glock_get(sdp, ip->i_no_addr, &gfs2_inode_glops, CREATE, &ip->i_gl);
+	error = gfs2_glock_get(sdp, ip->i_no_addr, &gfs2_user_inode_glops,
+			       CREATE, &ip->i_gl);
 	if (error)
 		goto fail_free_inode;
 	flush_delayed_work(&ip->i_gl->gl_work);
fs/gfs2/ops_fstype.c +11 -3
index 8a09f3aa609c..531c1870de8e 100644
@@ -431,7 +431,7 @@ static int init_locking(struct gfs2_sbd *sdp, struct gfs2_holder *mount_gh,
 }
 
 static int gfs2_lookup_root(struct super_block *sb, struct dentry **dptr,
-			    u64 no_addr, const char *name)
+			    u64 no_addr, const char *name, int journaled)
 {
 	struct gfs2_sbd *sdp = sb->s_fs_info;
 	struct dentry *dentry;
@@ -443,6 +443,11 @@ static int gfs2_lookup_root(struct super_block *sb, struct dentry **dptr,
 		fs_err(sdp, "can't read in %s inode: %ld\n", name, PTR_ERR(inode));
 		return PTR_ERR(inode);
 	}
+	if (!journaled) {
+		struct gfs2_inode *ip = GFS2_I(inode);
+
+		ip->i_gl->gl_ops = &gfs2_sys_inode_glops;
+	}
 	dentry = d_make_root(inode);
 	if (!dentry) {
 		fs_err(sdp, "can't alloc %s dentry\n", name);
@@ -491,13 +496,13 @@ static int init_sb(struct gfs2_sbd *sdp, int silent)
 
 	/* Get the root inode */
 	no_addr = sdp->sd_sb.sb_root_dir.no_addr;
-	ret = gfs2_lookup_root(sb, &sdp->sd_root_dir, no_addr, "root");
+	ret = gfs2_lookup_root(sb, &sdp->sd_root_dir, no_addr, "root", 1);
 	if (ret)
 		goto out;
 
 	/* Get the master inode */
 	no_addr = sdp->sd_sb.sb_master_dir.no_addr;
-	ret = gfs2_lookup_root(sb, &sdp->sd_master_dir, no_addr, "master");
+	ret = gfs2_lookup_root(sb, &sdp->sd_master_dir, no_addr, "master", 0);
 	if (ret) {
 		dput(sdp->sd_root_dir);
 		goto out;
@@ -532,6 +537,7 @@ static void gfs2_others_may_mount(struct gfs2_sbd *sdp)
 static int gfs2_jindex_hold(struct gfs2_sbd *sdp, struct gfs2_holder *ji_gh)
 {
 	struct gfs2_inode *dip = GFS2_I(sdp->sd_jindex);
+	struct gfs2_inode *ip;
 	struct qstr name;
 	char buf[20];
 	struct gfs2_jdesc *jd;
@@ -579,6 +585,8 @@ static int gfs2_jindex_hold(struct gfs2_sbd *sdp, struct gfs2_holder *ji_gh)
 			break;
 		}
 
+		ip = GFS2_I(jd->jd_inode);
+		ip->i_gl->gl_ops = &gfs2_sys_inode_glops;
 		spin_lock(&sdp->sd_jindex_spin);
 		jd->jd_jid = sdp->sd_journals++;
 		list_add_tail(&jd->jd_list, &sdp->sd_jindex_list);
fs/gfs2/sys.c +10 -2
index b772ff5509bb..dd32e266fd2e 100644
@@ -264,8 +264,16 @@ static ssize_t demote_rq_store(struct gfs2_sbd *sdp, const char *buf, size_t len
 	if (!test_and_set_bit(SDF_DEMOTE, &sdp->sd_flags))
 		fs_info(sdp, "demote interface used\n");
 	rv = gfs2_glock_get(sdp, glnum, glops, 0, &gl);
-	if (rv)
-		return rv;
+	if (rv) {
+		/* If the glock is not found and it's supposed to be an inode,
+		 * check if it's one of the system inodes. */
+		if (gltype == LM_TYPE_INODE) {
+			glops = &gfs2_sys_inode_glops;
+			rv = gfs2_glock_get(sdp, glnum, glops, 0, &gl);
+			if (rv)
+				return rv;
+		}
+	}
 	gfs2_glock_cb(gl, glmode);
 	gfs2_glock_put(gl);
 	return len;
Signature · 1 line
2.20.1
#15 lore raw

PATCH v3 12/19 gfs2: Don't loop forever in gfs2_freeze if withdrawn

In reply to Bob Peterson

Before this patch, function gfs2_freeze would loop forever if the file system trying to be frozen is withdrawn. That's because function gfs2_lock_fs_check_clean tries to enqueue the glock of the journal and the gfs2_glock returns -EIO because you can't enqueue a journaled glock after a withdraw.

This patch moves the check for file system withdraw inside the loop so that the loop can end when withdraw occurs.

---
 fs/gfs2/super.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
fs/gfs2/super.c +5 -5
index 378519beb6c3..4ae23c338034 100644
@@ -1014,12 +1014,12 @@ static int gfs2_freeze(struct super_block *sb)
 	if (atomic_read(&sdp->sd_freeze_state) != SFS_UNFROZEN)
 		goto out;
 
-	if (gfs2_withdrawn(sdp)) {
-		error = -EINVAL;
-		goto out;
-	}
-
 	for (;;) {
+		if (gfs2_withdrawn(sdp)) {
+			error = -EINVAL;
+			goto out;
+		}
+
 		error = gfs2_lock_fs_check_clean(sdp, &sdp->sd_freeze_gh);
 		if (!error)
 			break;
Signature · 1 line
2.20.1
#16 lore raw

PATCH v3 13/19 gfs2: Make secondary withdrawers wait for first withdrawer

In reply to Bob Peterson

Before this patch, if a process encountered an error and decided to withdraw, if another process was already in the process of withdrawing, the secondary withdraw would be silently ignored, which set it free to proceed with its processing, unlock any locks, etc. That's correct behavior if the original withdrawer encounters further errors down the road. However, second withdrawers need to wait for the first withdrawer to finish its withdraw before proceeding. If we don't wait we could end up assuming everything is alright, unlock glocks and telling other nodes they can have the glock, despite the fact that a withdraw is still ongoing and may require a journal replay before any locks are released. For example, if an rgrp glock is freed by a process that didn't wait for the withdraw, a journal replay could introduce file system corruption by replaying a rgrp block that has already been granted to another node.

This patch makes secondary withdrawers wait until the primary withdrawer is finished with its processing before proceeding.

---
 fs/gfs2/incore.h |  3 +++
 fs/gfs2/util.c   | 21 +++++++++++++++++++--
 2 files changed, 22 insertions(+), 2 deletions(-)
fs/gfs2/incore.h +3 -0
index b16b82e5b85a..0b283bbd20ea 100644
@@ -628,6 +628,7 @@ enum {
 	SDF_SKIP_DLM_UNLOCK	= 8,
 	SDF_FORCE_AIL_FLUSH     = 9,
 	SDF_WITHDRAW		= 10, /* Will withdraw eventually */
+	SDF_WITHDRAW_IN_PROG	= 11, /* Withdraw is in progress */
 };
 
 enum gfs2_freeze_state {
@@ -837,6 +838,8 @@ struct gfs2_sbd {
 	struct bio *sd_log_bio;
 	wait_queue_head_t sd_log_flush_wait;
 	int sd_log_error; /* First log error */
+	atomic_t sd_withdrawer;
+	wait_queue_head_t sd_withdraw_wait;
 
 	atomic_t sd_reserving_log;
 	wait_queue_head_t sd_reserving_log_wait;
fs/gfs2/util.c +19 -2
index 398d3cedb1a4..ad56f934ff0f 100644
@@ -89,9 +89,23 @@ int gfs2_lm_withdraw(struct gfs2_sbd *sdp, const char *fmt, ...)
 	struct va_format vaf;
 
 	if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW &&
-	    test_and_set_bit(SDF_SHUTDOWN, &sdp->sd_flags))
-		return 0;
+	    test_and_set_bit(SDF_SHUTDOWN, &sdp->sd_flags)) {
+		if (!test_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags))
+			return -1;
+
+		fs_warn(sdp, "Pid %d waiting for process %d to withdraw.\n",
+			pid_nr(task_pid(current)),
+			atomic_read(&sdp->sd_withdrawer));
+		wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_IN_PROG,
+			    TASK_UNINTERRUPTIBLE);
+		fs_warn(sdp, "Pid %d done waiting for process %d.\n",
+			pid_nr(task_pid(current)),
+			atomic_read(&sdp->sd_withdrawer));
+		return -1;
+	}
 
+	set_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags);
+	atomic_set(&sdp->sd_withdrawer, pid_nr(task_pid(current)));
 	if (fmt) {
 		va_start(args, fmt);
 
@@ -119,6 +133,9 @@ int gfs2_lm_withdraw(struct gfs2_sbd *sdp, const char *fmt, ...)
 		set_bit(SDF_SKIP_DLM_UNLOCK, &sdp->sd_flags);
 		fs_err(sdp, "withdrawn\n");
 		dump_stack();
+		clear_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags);
+		smp_mb__after_atomic();
+		wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_IN_PROG);
 	}
 
 	if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
Signature · 1 line
2.20.1
#17 lore raw

PATCH v3 14/19 gfs2: Don't write log headers after file system withdraw

In reply to Bob Peterson

Before this patch, when a node withdrew a gfs2 file system, it wrote a (clean) unmount log header. That's wrong. You don't want to write anything to the journal once you're withdrawn because that's acknowledging that the transaction is complete and the journal is in good shape, neither of which may be a valid assumption when the file system is withdrawn. This is especially true if the withdraw was caused due to io errors writing to the journal in the first place. The best course of action is to leave the journal "as is" until it may be safely replayed during journal recovery, regardless of whether it's done by this node or another.

---
 fs/gfs2/log.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
fs/gfs2/log.c +6 -1
index 6169276aa9e6..771589b2e225 100644
@@ -693,12 +693,16 @@ void gfs2_write_log_header(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd,
 {
 	struct gfs2_log_header *lh;
 	u32 hash, crc;
-	struct page *page = mempool_alloc(gfs2_page_pool, GFP_NOIO);
+	struct page *page;
 	struct gfs2_statfs_change_host *l_sc = &sdp->sd_statfs_local;
 	struct timespec64 tv;
 	struct super_block *sb = sdp->sd_vfs;
 	u64 dblock;
 
+	if (gfs2_withdrawn(sdp))
+		goto out;
+
+	page = mempool_alloc(gfs2_page_pool, GFP_NOIO);
 	lh = page_address(page);
 	clear_page(lh);
 
@@ -751,6 +755,7 @@ void gfs2_write_log_header(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd,
 
 	gfs2_log_write(sdp, page, sb->s_blocksize, 0, dblock);
 	gfs2_log_submit_bio(&sdp->sd_log_bio, REQ_OP_WRITE, op_flags);
+out:
 	log_flush_wait(sdp);
 }
 
Signature · 1 line
2.20.1
#18 lore raw

PATCH v3 15/19 gfs2: Force withdraw to replay journals and wait for it to finish

In reply to Bob Peterson

When a node withdraws from a file system, it often leaves its journal in an incomplete state. This is especially true when the withdraw is caused by io errors writing to the journal. Before this patch, a withdraw would try to write a "shutdown" record to the journal, tell dlm it's done with the file system, and none of the other nodes know about the problem. Later, when the problem is fixed and the withdrawn node is rebooted, it would then discover that its own journal was incomplete, and replay it. However, replaying it at this point is almost guaranteed to introduce corruption because the other nodes are likely to have used affected resource groups that appeared in the journal since the time of the withdraw. Replaying the journal later will overwrite any changes made, and not through any fault of dlm, which was instructed during the withdraw to release those resources.

This patch makes file system withdraws seen by the entire cluster. Withdrawing nodes dequeue their journal glock to allow recovery.

The remaining nodes check all the journals to see if they are clean or in need of replay. They try to replay dirty journals, but only the journals of withdrawn nodes will be "not busy" and therefore available for replay.

Until the journal replay is complete, no i/o related glocks may be given out, to ensure that the replay does not cause the aforementioned corruption: We cannot allow any journal replay to overwrite blocks associated with a glock once it is held. The glocks not affected by a withdraw are permitted to be passed around as normal during a withdraw. A new glops flag, called GLOF_OK_AT_WITHDRAW, indicates glocks that may be passed around freely while a withdraw is taking place.

One such glock is the "live" glock which is now used to signal when a withdraw occurs. When a withdraw occurs, the node signals its withdraw by dequeueing the "live" glock and trying to enqueue it in EX mode, thus forcing the other nodes to all see a demote request, by way of a "1CB" (one callback) try lock. The "live" glock is not granted in EX; the callback is only just used to indicate a withdraw has occurred.

Note that all nodes in the cluster must wait for the recovering node to finish replaying the withdrawing node's journal before continuing. To this end, it checks that the journals are clean multiple times in a retry loop.

---
 fs/gfs2/glock.c      |  32 ++++++++++--
 fs/gfs2/glock.h      |   1 +
 fs/gfs2/glops.c      |  56 +++++++++++++++++++++
 fs/gfs2/incore.h     |   5 ++
 fs/gfs2/lock_dlm.c   |  32 ++++++++++++
 fs/gfs2/log.c        |   1 +
 fs/gfs2/meta_io.c    |   2 +-
 fs/gfs2/ops_fstype.c |   4 +-
 fs/gfs2/quota.c      |   4 ++
 fs/gfs2/super.c      |  47 ++++++++++++------
 fs/gfs2/super.h      |   1 +
 fs/gfs2/util.c       | 116 ++++++++++++++++++++++++++++++++++++++++++-
 12 files changed, 281 insertions(+), 20 deletions(-)
fs/gfs2/glock.c +29 -3
index eeb84d8d21f3..3d1b9bdfd0de 100644
@@ -247,7 +247,7 @@ static void __gfs2_glock_put(struct gfs2_glock *gl)
 	gfs2_glock_remove_from_lru(gl);
 	spin_unlock(&gl->gl_lockref.lock);
 	GLOCK_BUG_ON(gl, !list_empty(&gl->gl_holders));
-	GLOCK_BUG_ON(gl, mapping && mapping->nrpages);
+	GLOCK_BUG_ON(gl, mapping && mapping->nrpages && !gfs2_withdrawn(sdp));
 	trace_gfs2_glock_put(gl);
 	sdp->sd_lockstruct.ls_ops->lm_put_lock(gl);
 }
@@ -548,7 +548,9 @@ __acquires(&gl->gl_lockref.lock)
 	int ret;
 
 	if (unlikely(gfs2_withdrawn(sdp)) &&
-	    (glops->go_flags & GLOF_JOURNALED) && target != LM_ST_UNLOCKED)
+	    (glops->go_flags & GLOF_JOURNALED) &&
+	    (gh && !(LM_FLAG_NOEXP & gh->gh_flags)) &&
+	    target != LM_ST_UNLOCKED)
 		return;
 	lck_flags &= (LM_FLAG_TRY | LM_FLAG_TRY_1CB | LM_FLAG_NOEXP |
 		      LM_FLAG_PRIORITY);
@@ -1096,7 +1098,7 @@ int gfs2_glock_nq(struct gfs2_holder *gh)
 	struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
 	int error = 0;
 
-	if (unlikely(gfs2_withdrawn(sdp)) &&
+	if (unlikely(gfs2_withdrawn(sdp)) && !(LM_FLAG_NOEXP & gh->gh_flags) &&
 	    (gl->gl_ops->go_flags & GLOF_JOURNALED))
 		return -EIO;
 
@@ -1141,11 +1143,30 @@ int gfs2_glock_poll(struct gfs2_holder *gh)
 void gfs2_glock_dq(struct gfs2_holder *gh)
 {
 	struct gfs2_glock *gl = gh->gh_gl;
+	struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
 	const struct gfs2_glock_operations *glops = gl->gl_ops;
 	unsigned delay = 0;
 	int fast_path = 0;
 
 	spin_lock(&gl->gl_lockref.lock);
+	/**
+	 * If we're in the process of file system withdraw, we cannot just
+	 * dequeue any glocks until our journal is recovered, lest we
+	 * introduce file system corruption. We need two exceptions to this
+	 * rule: We need to allow unlocking of nondisk glocks and the glock
+	 * for our own journal that needs recovery. Note that evict may also
+	 * call this for the journal inode, but without using sdp->sd_jinode_gh
+	 * so we need to check the glock, not just the holder of the glock.
+	 */
+	if (test_bit(SDF_SHUTDOWN, &sdp->sd_flags) &&
+	    test_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags) &&
+	    (gl->gl_ops->go_flags & GLOF_JOURNALED) &&
+	    gh->gh_gl != sdp->sd_jinode_gl) {
+		sdp->sd_glock_dqs_held++;
+		might_sleep();
+		wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_RECOVERY,
+			    TASK_UNINTERRUPTIBLE);
+	}
 	if (gh->gh_flags & GL_NOCACHE)
 		handle_callback(gl, LM_ST_UNLOCKED, 0, false);
 
@@ -1625,6 +1646,11 @@ static void dump_glock_func(struct gfs2_glock *gl)
 	dump_glock(NULL, gl);
 }
 
+void gfs2_gl_flushwork(struct gfs2_sbd *sdp)
+{
+	flush_workqueue(glock_workqueue);
+}
+
 /**
  * gfs2_gl_hash_clear - Empty out the glock hash table
  * @sdp: the filesystem
fs/gfs2/glock.h +1 -0
index 936b3295839c..c1c40e2dbd96 100644
@@ -202,6 +202,7 @@ extern int gfs2_glock_nq_num(struct gfs2_sbd *sdp, u64 number,
 			     struct gfs2_holder *gh);
 extern int gfs2_glock_nq_m(unsigned int num_gh, struct gfs2_holder *ghs);
 extern void gfs2_glock_dq_m(unsigned int num_gh, struct gfs2_holder *ghs);
+extern void gfs2_gl_flushwork(struct gfs2_sbd *sdp);
 extern void gfs2_dump_glock(struct seq_file *seq, struct gfs2_glock *gl);
 #define GLOCK_BUG_ON(gl,x) do { if (unlikely(x)) { gfs2_dump_glock(NULL, gl); BUG(); } } while(0)
 extern __printf(2, 3)
fs/gfs2/glops.c +56 -0

80 lines not on this page · raw message

fs/gfs2/incore.h +5 -0
index 0b283bbd20ea..60b094bad47e 100644
@@ -629,6 +629,9 @@ enum {
 	SDF_FORCE_AIL_FLUSH     = 9,
 	SDF_WITHDRAW		= 10, /* Will withdraw eventually */
 	SDF_WITHDRAW_IN_PROG	= 11, /* Withdraw is in progress */
+	SDF_REMOTE_WITHDRAW	= 12, /* Performing remote recovery */
+	SDF_WITHDRAW_RECOVERY	= 13, /* Wait for journal recovery when we are
+					 withdrawing */
 };
 
 enum gfs2_freeze_state {
@@ -777,6 +780,7 @@ struct gfs2_sbd {
 	struct gfs2_jdesc *sd_jdesc;
 	struct gfs2_holder sd_journal_gh;
 	struct gfs2_holder sd_jinode_gh;
+	struct gfs2_glock *sd_jinode_gl;
 
 	struct gfs2_holder sd_sc_gh;
 	struct gfs2_holder sd_qc_gh;
@@ -863,6 +867,7 @@ struct gfs2_sbd {
 
 	unsigned long sd_last_warning;
 	struct dentry *debugfs_dir;    /* debugfs directory */
+	unsigned long sd_glock_dqs_held;
 };
 
 static inline void gfs2_glstats_inc(struct gfs2_glock *gl, int which)
fs/gfs2/lock_dlm.c +32 -0
index 9329f86ffcbe..444c481f985d 100644
@@ -19,6 +19,8 @@
 
 #include "incore.h"
 #include "glock.h"
+#include "glops.h"
+#include "recovery.h"
 #include "util.h"
 #include "sys.h"
 #include "trace_gfs2.h"
@@ -325,6 +327,7 @@ static void gdlm_cancel(struct gfs2_glock *gl)
 /*
  * dlm/gfs2 recovery coordination using dlm_recover callbacks
  *
+ *  0. gfs2 checks for another cluster node withdraw, needing journal replay
  *  1. dlm_controld sees lockspace members change
  *  2. dlm_controld blocks dlm-kernel locking activity
  *  3. dlm_controld within dlm-kernel notifies gfs2 (recover_prep)
@@ -573,6 +576,28 @@ static int control_lock(struct gfs2_sbd *sdp, int mode, uint32_t flags)
 			 &ls->ls_control_lksb, "control_lock");
 }
 
+/**
+ * remote_withdraw - react to a node withdrawing from the file system
+ * @sdp: The superblock
+ */
+static void remote_withdraw(struct gfs2_sbd *sdp)
+{
+	struct gfs2_jdesc *jd;
+	int ret = 0, count = 0;
+
+	list_for_each_entry(jd, &sdp->sd_jindex_list, jd_list) {
+		if (jd->jd_jid == sdp->sd_lockstruct.ls_jid)
+			continue;
+		ret = gfs2_recover_journal(jd, true);
+		if (ret)
+			break;
+		count++;
+	}
+
+	/* Now drop the additional reference we acquired */
+	fs_err(sdp, "Journals checked: %d, ret = %d.\n", count, ret);
+}
+
 static void gfs2_control_func(struct work_struct *work)
 {
 	struct gfs2_sbd *sdp = container_of(work, struct gfs2_sbd, sd_control_work.work);
@@ -583,6 +608,13 @@ static void gfs2_control_func(struct work_struct *work)
 	int recover_size;
 	int i, error;
 
+	/* First check for other nodes that may have done a withdraw. */
+	if (test_bit(SDF_REMOTE_WITHDRAW, &sdp->sd_flags)) {
+		remote_withdraw(sdp);
+		clear_bit(SDF_REMOTE_WITHDRAW, &sdp->sd_flags);
+		return;
+	}
+
 	spin_lock(&ls->ls_recover_spin);
 	/*
 	 * No MOUNT_DONE means we're still mounting; control_mount()
fs/gfs2/log.c +1 -0
index 771589b2e225..732b585f33be 100644
@@ -1023,6 +1023,7 @@ int gfs2_logd(void *data)
 					 "withdrawing the file system to "
 					 "prevent further damage.\n",
 					 sdp->sd_fsname, sdp->sd_log_error);
+			break;
 		}
 
 		did_flush = false;
fs/gfs2/meta_io.c +1 -1
index b125008f064f..49adeadd5868 100644
@@ -255,7 +255,7 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
 	struct buffer_head *bh, *bhs[2];
 	int num = 0;
 
-	if (unlikely(gfs2_withdrawn(sdp))) {
+	if (unlikely(gfs2_withdrawn(sdp)) && gl != sdp->sd_jinode_gl) {
 		*bhp = NULL;
 		return -EIO;
 	}
fs/gfs2/ops_fstype.c +3 -1
index 531c1870de8e..c4222c1fa735 100644
@@ -650,7 +650,8 @@ static int init_journal(struct gfs2_sbd *sdp, int undo)
 
 		error = gfs2_glock_nq_num(sdp, sdp->sd_lockstruct.ls_jid,
 					  &gfs2_journal_glops,
-					  LM_ST_EXCLUSIVE, LM_FLAG_NOEXP,
+					  LM_ST_EXCLUSIVE,
+					  LM_FLAG_NOEXP | GL_NOCACHE,
 					  &sdp->sd_journal_gh);
 		if (error) {
 			fs_err(sdp, "can't acquire journal glock: %d\n", error);
@@ -658,6 +659,7 @@ static int init_journal(struct gfs2_sbd *sdp, int undo)
 		}
 
 		ip = GFS2_I(sdp->sd_jdesc->jd_inode);
+		sdp->sd_jinode_gl = ip->i_gl;
 		error = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED,
 					   LM_FLAG_NOEXP | GL_EXACT | GL_NOCACHE,
 					   &sdp->sd_jinode_gh);
fs/gfs2/quota.c +4 -0
index acb6cbdd1431..6075d26c9d78 100644
@@ -1575,6 +1575,10 @@ int gfs2_quotad(void *data)
 		else
 			t = 0;
 		finish_wait(&sdp->sd_quota_wait, &wait);
+
+		if (atomic_read(&sdp->sd_withdrawer) ==
+		    pid_nr(task_pid(current)))
+			break;
 	}
 
 	return 0;
fs/gfs2/super.c +33 -14

90 lines not on this page · raw message

fs/gfs2/super.h +1 -0
index 73c97dccae21..e859c6d5bb3e 100644
@@ -45,6 +45,7 @@ extern void gfs2_statfs_change_in(struct gfs2_statfs_change_host *sc,
 extern void update_statfs(struct gfs2_sbd *sdp, struct buffer_head *m_bh,
 			  struct buffer_head *l_bh);
 extern int gfs2_statfs_sync(struct super_block *sb, int type);
+extern int gfs2_make_fs_ro(struct gfs2_sbd *sdp, int withdrawing);
 extern void gfs2_freeze_func(struct work_struct *work);
 
 extern struct file_system_type gfs2_fs_type;
fs/gfs2/util.c +115 -1

145 lines not on this page · raw message

Signature · 1 line
2.20.1
#19 lore raw

PATCH v3 16/19 gfs2: simply gfs2_freeze by removing case

In reply to Bob Peterson

Function gfs2_freeze had a case statement that simply checked the error code, but the break statements just made the logic hard to read. This patch simplifies the logic in favor of a simple if.

---
 fs/gfs2/super.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
fs/gfs2/super.c +2 -8
index 1582cb11866e..7ffeb4379222 100644
@@ -1043,20 +1043,14 @@ static int gfs2_freeze(struct super_block *sb)
 		if (!error)
 			break;
 
-		switch (error) {
-		case -EBUSY:
+		if (error == -EBUSY)
 			fs_err(sdp, "waiting for recovery before freeze\n");
-			break;
-
-		default:
+		else
 			fs_err(sdp, "error freezing FS: %d\n", error);
-			break;
-		}
 
 		fs_err(sdp, "retrying...\n");
 		msleep(1000);
 	}
-	error = 0;
 out:
 	mutex_unlock(&sdp->sd_freeze_mutex);
 	return error;
Signature · 1 line
2.20.1
#20 lore raw

PATCH v3 17/19 gfs2: Add verbose option to check_journal_clean

In reply to Bob Peterson

Before this patch, function check_journal_clean would give messages related to journal recovery. That's fine for mount time, but when a node withdraws and forces replay that way, we don't want all those distracting and misleading messages. This patch adds a new parameter to make those messages optional.

---
 fs/gfs2/ops_fstype.c |  2 +-
 fs/gfs2/util.c       | 23 ++++++++++++++++-------
 fs/gfs2/util.h       |  4 +++-
 3 files changed, 20 insertions(+), 9 deletions(-)
fs/gfs2/ops_fstype.c +1 -1
index c4222c1fa735..d8bca92f34a1 100644
@@ -690,7 +690,7 @@ static int init_journal(struct gfs2_sbd *sdp, int undo)
 			struct gfs2_jdesc *jd = gfs2_jdesc_find(sdp, x);
 
 			if (sdp->sd_args.ar_spectator) {
-				error = check_journal_clean(sdp, jd);
+				error = check_journal_clean(sdp, jd, true);
 				if (error)
 					goto fail_jinode_gh;
 				continue;
fs/gfs2/util.c +16 -7
index 5e39859d58d8..2dd3fb8975a7 100644
@@ -48,7 +48,8 @@ void gfs2_assert_i(struct gfs2_sbd *sdp)
  *
  * Returns: 0 if the journal is clean or locked, else an error
  */
-int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd)
+int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd,
+			bool verbose)
 {
 	int error;
 	struct gfs2_holder j_gh;
@@ -59,23 +60,31 @@ int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd)
 	error = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, LM_FLAG_NOEXP |
 				   GL_EXACT | GL_NOCACHE, &j_gh);
 	if (error) {
-		fs_err(sdp, "Error locking journal for spectator mount.\n");
+		if (verbose)
+			fs_err(sdp, "Error %d locking journal for spectator "
+			       "mount.\n", error);
 		return -EPERM;
 	}
 	error = gfs2_jdesc_check(jd);
 	if (error) {
-		fs_err(sdp, "Error checking journal for spectator mount.\n");
+		if (verbose)
+			fs_err(sdp, "Error checking journal for spectator "
+			       "mount.\n");
 		goto out_unlock;
 	}
 	error = gfs2_find_jhead(jd, &head);
 	if (error) {
-		fs_err(sdp, "Error parsing journal for spectator mount.\n");
+		if (verbose)
+			fs_err(sdp, "Error parsing journal for spectator "
+			       "mount.\n");
 		goto out_unlock;
 	}
 	if (!(head.lh_flags & GFS2_LOG_HEAD_UNMOUNT)) {
 		error = -EPERM;
-		fs_err(sdp, "jid=%u: Journal is dirty, so the first mounter "
-		       "must not be a spectator.\n", jd->jd_jid);
+		if (verbose)
+			fs_err(sdp, "jid=%u: Journal is dirty, so the first "
+			       "mounter must not be a spectator.\n",
+			       jd->jd_jid);
 	}
 
 out_unlock:
@@ -174,7 +183,7 @@ static void signal_our_withdraw(struct gfs2_sbd *sdp)
 
 	/* Now wait until recovery complete. */
 	for (tries = 0; tries < 10; tries++) {
-		ret = check_journal_clean(sdp, sdp->sd_jdesc);
+		ret = check_journal_clean(sdp, sdp->sd_jdesc, false);
 		if (!ret)
 			break;
 		msleep(HZ);
fs/gfs2/util.h +3 -1
index 49d46865dceb..345b906a6142 100644
@@ -131,7 +131,9 @@ static inline void gfs2_metatype_set(struct buffer_head *bh, u16 type,
 
 int gfs2_io_error_i(struct gfs2_sbd *sdp, const char *function,
 		    char *file, unsigned int line);
-int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd);
+
+extern int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd,
+			       bool verbose);
 
 #define gfs2_io_error(sdp) \
 gfs2_io_error_i((sdp), __func__, __FILE__, __LINE__);
Signature · 1 line
2.20.1
#21 lore raw

PATCH v3 18/19 gfs2: Check for log write errors before telling dlm to unlock

In reply to Bob Peterson

Before this patch, function do_xmote just assumed all the writes submitted to the journal were finished and successful, and it called the go_unlock function to release the dlm lock. But if they're not, and a revoke failed to make its way to the journal, a journal replay on another node will cause corruption if we let the go_inval function continue and tell dlm to release the glock to another node. This patch adds a couple checks for errors in do_xmote after the calls to go_sync and go_inval. If an error is found, we cannot withdraw yet, because the withdraw itself uses glocks to make the file system read-only. Instead, we flag the error. Later, asserts should cause another node to replay the journal before continuing, thus protecting rgrp and dinode glocks and maintaining the integrity of the metadata. Note that we only need to do this for journaled glocks. System glocks should be able to progress even under withdrawn conditions.

---
 fs/gfs2/glock.c | 44 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 41 insertions(+), 3 deletions(-)
fs/gfs2/glock.c +41 -3
index 3d1b9bdfd0de..f4129305a815 100644
@@ -570,8 +570,47 @@ __acquires(&gl->gl_lockref.lock)
 	spin_unlock(&gl->gl_lockref.lock);
 	if (glops->go_sync)
 		glops->go_sync(gl);
+	/**
+	 * Check for an error encountered since we called go_sync. If so, we
+	 * can't withdraw from the glock code because the withdraw code itself
+	 * uses glocks (see function signal_our_withdraw) to change the mount
+	 * to read-only. Most importantly, we must not call dlm to unlock the
+	 * glock until the journal is in a known good state (after journal
+	 * replay) otherwise other nodes may use the object (rgrp or dinode)
+	 * and then later, journal replay will corrupt the file system. The
+	 * best we can do now is wait for the logd daemon to see sd_log_error
+	 * and withdraw, and in the meantime, requeue the work for later.
+	 */
+	if ((glops->go_flags & GLOF_JOURNALED) && sdp->sd_log_error) {
+		gfs2_glock_hold(gl);
+		finish_xmote(gl, LM_OUT_ERROR | LM_ST_UNLOCKED);
+		gfs2_glock_queue_work(gl, GL_GLOCK_DFT_HOLD);
+		goto out;
+	}
 	if (test_bit(GLF_INVALIDATE_IN_PROGRESS, &gl->gl_flags))
 		glops->go_inval(gl, target == LM_ST_DEFERRED ? 0 : DIO_METADATA);
+	/**
+	 * If the go_inval left some items on the ail, that's a problem. We
+	 * need to withdraw, but we can't, as explained above. Warn about the
+	 * error, dump the glock, then fall through and wait for logd to do
+	 * the withdraw for us.
+	 */
+	if (atomic_read(&gl->gl_ail_count) != 0) {
+		if (!cmpxchg(&sdp->sd_log_error, 0, -EIO)) {
+			gfs2_assert_warn(sdp, !atomic_read(&gl->gl_ail_count));
+			gfs2_dump_glock(NULL, gl);
+		}
+	}
+	/**
+	 * Check for an error encountered while we were doing go_inval, and
+	 * handle it the same way we did for the go_sync case.
+	 */
+	if ((glops->go_flags & GLOF_JOURNALED) && sdp->sd_log_error) {
+		gfs2_glock_hold(gl);
+		finish_xmote(gl, LM_OUT_ERROR | LM_ST_UNLOCKED);
+		gfs2_glock_queue_work(gl, GL_GLOCK_DFT_HOLD);
+		goto out;
+	}
 	clear_bit(GLF_INVALIDATE_IN_PROGRESS, &gl->gl_flags);
 
 	gfs2_glock_hold(gl);
@@ -583,8 +622,7 @@ __acquires(&gl->gl_lockref.lock)
 		    test_bit(SDF_SKIP_DLM_UNLOCK, &sdp->sd_flags)) {
 			finish_xmote(gl, target);
 			gfs2_glock_queue_work(gl, 0);
-		}
-		else if (ret) {
+		} else if (ret) {
 			fs_err(sdp, "lm_lock ret %d\n", ret);
 			GLOCK_BUG_ON(gl, !gfs2_withdrawn(sdp));
 		}
@@ -592,7 +630,7 @@ __acquires(&gl->gl_lockref.lock)
 		finish_xmote(gl, target);
 		gfs2_glock_queue_work(gl, 0);
 	}
-
+out:
 	spin_lock(&gl->gl_lockref.lock);
 }
 
Signature · 1 line
2.20.1
#22 lore raw

PATCH v3 19/19 gfs2: Do log_flush in gfs2_ail_empty_gl even if ail list is empty

In reply to Bob Peterson

Before this patch, if gfs2_ail_empty_gl saw there was nothing on the ail list, it would return and not flush the log. The problem is that there could still be a revoke for the rgrp sitting on the sd_log_le_revoke list that's been recently taken off the ail list. But that revoke still needs to be written, and the rgrp_go_inval still needs to call log_flush_wait to ensure the revokes are all properly written to the journal before we relinquish control of the glock to another node. If we give the glock to another node before we have this knowledge, the node might crash and its journal replayed, in which case the missing revoke would allow the journal replay to replay the rgrp over top of the rgrp we already gave to another node, thus overwriting its changes and corrupting the file system.

This patch makes gfs2_ail_empty_gl still call gfs2_log_flush rather than returning.

---
 fs/gfs2/glops.c | 26 +++++++++++++++++++++++++-
 fs/gfs2/log.c   |  2 +-
 fs/gfs2/log.h   |  1 +
 3 files changed, 27 insertions(+), 2 deletions(-)
fs/gfs2/glops.c +25 -1
index ff50013c6aa9..1cd2a3c69d63 100644
@@ -93,8 +93,31 @@ static void gfs2_ail_empty_gl(struct gfs2_glock *gl)
 	INIT_LIST_HEAD(&tr.tr_databuf);
 	tr.tr_revokes = atomic_read(&gl->gl_ail_count);
 
-	if (!tr.tr_revokes)
+	if (!tr.tr_revokes) {
+		/**
+		 * We have nothing on the ail, but there could be revokes on
+		 * the sdp revoke queue, in which case, we still want to flush
+		 * the log and wait for it to finish.
+		 *
+		 * If the sdp revoke list is empty too, we might still have an
+		 * io outstanding for writing revokes, so we should wait for
+		 * it before returning.
+		 *
+		 * If none of these conditions are true, our revokes are all
+		 * flushed and we can return.
+		 */
+		gfs2_log_lock(sdp);
+		if (test_bit(GLF_REVOKES, &gl->gl_flags)) {
+			gfs2_log_unlock(sdp);
+			goto flush;
+		} else if (atomic_read(&sdp->sd_log_in_flight)) {
+			gfs2_log_unlock(sdp);
+			log_flush_wait(sdp);
+		} else {
+			gfs2_log_unlock(sdp);
+		}
 		return;
+	}
 
 	/* A shortened, inline version of gfs2_trans_begin()
          * tr->alloced is not set since the transaction structure is
@@ -109,6 +132,7 @@ static void gfs2_ail_empty_gl(struct gfs2_glock *gl)
 	__gfs2_ail_flush(gl, 0, tr.tr_revokes);
 
 	gfs2_trans_end(sdp);
+flush:
 	gfs2_log_flush(sdp, NULL, GFS2_LOG_HEAD_FLUSH_NORMAL |
 		       GFS2_LFC_AIL_EMPTY_GL);
 }
fs/gfs2/log.c +1 -1
index 732b585f33be..a6c83cdf4e1b 100644
@@ -522,7 +522,7 @@ static void log_pull_tail(struct gfs2_sbd *sdp, unsigned int new_tail)
 }
 
 
-static void log_flush_wait(struct gfs2_sbd *sdp)
+void log_flush_wait(struct gfs2_sbd *sdp)
 {
 	DEFINE_WAIT(wait);
 
fs/gfs2/log.h +1 -0
index 7a34a3234266..cd1581bff293 100644
@@ -76,6 +76,7 @@ extern void gfs2_log_flush(struct gfs2_sbd *sdp, struct gfs2_glock *gl,
 			   u32 type);
 extern void gfs2_log_commit(struct gfs2_sbd *sdp, struct gfs2_trans *trans);
 extern void gfs2_ail1_flush(struct gfs2_sbd *sdp, struct writeback_control *wbc);
+extern void log_flush_wait(struct gfs2_sbd *sdp);
 
 extern void gfs2_log_shutdown(struct gfs2_sbd *sdp);
 extern int gfs2_logd(void *data);
Signature · 1 line
2.20.1
#23 lore raw

v3 cover gfs2: misc recovery patch collection

In reply to Bob Peterson

Hi,

Quoted 14 lines · Bob Peterson

Here is version 3 of the patch set I posted on 23 April. It is revised based on bugs I found testing with xfstests.

This is a collection of patches I've been using to address the myriad of recovery problems I've found. I'm still finding them, so the battle is not won yet. I'm not convinced we need all of these but I thought I'd send them anyway and get feedback. Previously I sent out a version of the patch "gfs2: Force withdraw to replay journals and wait for it to finish" that was too big and complex. So I broke it up into four patches, starting with "move check_journal_clean to util.c for future use". So those four need to be a set. There aren't many other dependencies between patches, so the others could probably be taken or rejected individually. There are more patches I still need to perfect, but maybe a few of the safer ones can be pushed to for-next.

There is quite a lot of good stuff here, but it would be good to split it up and make it clearer what is bug fixes, and what are new features,

Steve.

Quoted 45 lines

Bob Peterson (19):
gfs2: kthread and remount improvements
gfs2: eliminate tr_num_revoke_rm
gfs2: log which portion of the journal is replayed
gfs2: Warn when a journal replay overwrites a rgrp with buffers
gfs2: Introduce concept of a pending withdraw
gfs2: log error reform
gfs2: Only complain the first time an io error occurs in quota or log
gfs2: Stop ail1 wait loop when withdrawn
gfs2: Ignore recovery attempts if gfs2 has io error or is withdrawn
gfs2: move check_journal_clean to util.c for future use
gfs2: Allow some glocks to be used during withdraw
gfs2: Don't loop forever in gfs2_freeze if withdrawn
gfs2: Make secondary withdrawers wait for first withdrawer
gfs2: Don't write log headers after file system withdraw
gfs2: Force withdraw to replay journals and wait for it to finish
gfs2: simply gfs2_freeze by removing case
gfs2: Add verbose option to check_journal_clean
gfs2: Check for log write errors before telling dlm to unlock
gfs2: Do log_flush in gfs2_ail_empty_gl even if ail list is empty

  fs/gfs2/aops.c       |   4 +-
  fs/gfs2/file.c       |   2 +-
  fs/gfs2/glock.c      |  82 +++++++++++++++--
  fs/gfs2/glock.h      |   1 +
  fs/gfs2/glops.c      | 100 +++++++++++++++++++--
  fs/gfs2/glops.h      |   3 +-
  fs/gfs2/incore.h     |  21 ++++-
  fs/gfs2/inode.c      |  14 ++-
  fs/gfs2/lock_dlm.c   |  50 +++++++++++
  fs/gfs2/log.c        |  55 +++++++-----
  fs/gfs2/log.h        |   1 +
  fs/gfs2/lops.c       |  27 +++++-
  fs/gfs2/meta_io.c    |   6 +-
  fs/gfs2/ops_fstype.c |  65 ++++----------
  fs/gfs2/quota.c      |  10 ++-
  fs/gfs2/recovery.c   |   3 +-
  fs/gfs2/super.c      |  88 +++++++++++-------
  fs/gfs2/super.h      |   1 +
  fs/gfs2/sys.c        |  14 ++-
  fs/gfs2/trans.c      |   6 +-
  fs/gfs2/util.c       | 206 ++++++++++++++++++++++++++++++++++++++++---
  fs/gfs2/util.h       |  15 ++++
  22 files changed, 617 insertions(+), 157 deletions(-)