Skip to contentSkip to conversation

jbd2: Avoid infinite transaction commit loop

PATCH New Jan Kara 5 participants 17 messages Jun 24, 2024 – Jul 11, 2024

Subsystems (2)
lore
Share & tools
lore

Jan Kara

author
#0 lore raw

cover jbd2: Avoid infinite transaction commit loop

Hello,

Alexander has reported [1] that when he tries to online-resize a small filesystem JBD2 eventually BUGs in transaction commit code. This is caused by online resize code starting a transaction of size close to maximum allowed transaction size for the journal. When descriptor blocks are added to the transaction it actually exceeds maximum transaction size and that confuses start_this_handle() which enters infinite transaction commit loop (see patch 3 for details). This patch series fixes the confusion in start_this_handle(). There's still open question how to make online resizing with tiny journal work.

Honza

[1] https://lore.kernel.org/all/CA+hUFcuGs04JHZ_WzA1zGN57+ehL2qmHOt5a7RMpo+rv6Vyxtw@mail.gmail.com

author
#1 lore raw

PATCH v2 1/4 jbd2: Make jbd2_journal_get_max_txn_bufs() internal

There's no reason to have jbd2_journal_get_max_txn_bufs() public function. Currently all users are internal and can use journal->j_max_transaction_buffers instead. This saves some unnecessary recomputations of the limit as a bonus which becomes important as this function gets more complex in the following patch.

---
 fs/jbd2/commit.c     | 2 +-
 fs/jbd2/journal.c    | 5 +++++
 include/linux/jbd2.h | 5 -----
 3 files changed, 6 insertions(+), 6 deletions(-)
fs/jbd2/commit.c +1 -1
index 75ea4e9a5cab..e7fc912693bd 100644
@@ -766,7 +766,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
 		if (first_block < journal->j_tail)
 			freed += journal->j_last - journal->j_first;
 		/* Update tail only if we free significant amount of space */
-		if (freed < jbd2_journal_get_max_txn_bufs(journal))
+		if (freed < journal->j_max_transaction_buffers)
 			update_tail = 0;
 	}
 	J_ASSERT(commit_transaction->t_state == T_COMMIT);
fs/jbd2/journal.c +5 -0
index 03c4b9214f56..1bb73750d307 100644
@@ -1698,6 +1698,11 @@ journal_t *jbd2_journal_init_inode(struct inode *inode)
 	return journal;
 }
 
+static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
+{
+	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
+}
+
 /*
  * Given a journal_t structure, initialise the various fields for
  * startup of a new journaling session.  We use this both when creating
include/linux/jbd2.h +0 -5
index ab04c1c27fae..f91b930abe20 100644
@@ -1660,11 +1660,6 @@ int jbd2_wait_inode_data(journal_t *journal, struct jbd2_inode *jinode);
 int jbd2_fc_wait_bufs(journal_t *journal, int num_blks);
 int jbd2_fc_release_bufs(journal_t *journal);
 
-static inline int jbd2_journal_get_max_txn_bufs(journal_t *journal)
-{
-	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
-}
-
 /*
  * is_journal_abort
  *
Signature · 1 line
2.35.3
Reviewed-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] jbd2: Make jbd2_journal_get_max_txn_bufs() internal

Quoted 8 lines · Jan Kara

There's no reason to have jbd2_journal_get_max_txn_bufs() public function. Currently all users are internal and can use journal->j_max_transaction_buffers instead. This saves some unnecessary recomputations of the limit as a bonus which becomes important as this function gets more complex in the following patch.

A good cleanup.

Quoted 52 lines
---
 fs/jbd2/commit.c     | 2 +-
 fs/jbd2/journal.c    | 5 +++++
 include/linux/jbd2.h | 5 -----
 3 files changed, 6 insertions(+), 6 deletions(-)
fs/jbd2/commit.c +1 -1
index 75ea4e9a5cab..e7fc912693bd 100644
@@ -766,7 +766,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
 		if (first_block < journal->j_tail)
 			freed += journal->j_last - journal->j_first;
 		/* Update tail only if we free significant amount of space */
-		if (freed < jbd2_journal_get_max_txn_bufs(journal))
+		if (freed < journal->j_max_transaction_buffers)
 			update_tail = 0;
 	}
 	J_ASSERT(commit_transaction->t_state == T_COMMIT);
fs/jbd2/journal.c +5 -0
index 03c4b9214f56..1bb73750d307 100644
@@ -1698,6 +1698,11 @@ journal_t *jbd2_journal_init_inode(struct inode *inode)
 	return journal;
 }
 
+static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
+{
+	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
+}
+
 /*
  * Given a journal_t structure, initialise the various fields for
  * startup of a new journaling session.  We use this both when creating
include/linux/jbd2.h +0 -5
index ab04c1c27fae..f91b930abe20 100644
@@ -1660,11 +1660,6 @@ int jbd2_wait_inode_data(journal_t *journal, struct jbd2_inode *jinode);
 int jbd2_fc_wait_bufs(journal_t *journal, int num_blks);
 int jbd2_fc_release_bufs(journal_t *journal);
 
-static inline int jbd2_journal_get_max_txn_bufs(journal_t *journal)
-{
-	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
-}
-
 /*
  * is_journal_abort
  *
maintainer Applied
#3 lore raw

Re: [PATCH v2 1/4] jbd2: Make jbd2_journal_get_max_txn_bufs() internal

In reply to Jan Kara
Quoted 8 lines · Jan Kara

There's no reason to have jbd2_journal_get_max_txn_bufs() public function. Currently all users are internal and can use journal->j_max_transaction_buffers instead. This saves some unnecessary recomputations of the limit as a bonus which becomes important as this function gets more complex in the following patch.

[...]

Applied, thanks!

[1/4] jbd2: Make jbd2_journal_get_max_txn_bufs() internal

      commit: 4aa99c71e42ad60178c1154ec24e3df9c684fb67
[2/4] jbd2: Precompute number of transaction descriptor blocks
      commit: e3a00a23781c1f2fcda98a7aecaac515558e7a35
[3/4] jbd2: Avoid infinite transaction commit loop
      commit: 27ba5b67312a944576addc4df44ac3b709aabede
[4/4] jbd2: Drop pointless shrinker batch initialization
      commit: 1cf5b024a3ffa479ed14e520f81549fce037f322

Best regards,

Signature · 1 line
Theodore Ts'o <[email protected]>
author
#4 lore raw

PATCH v2 2/4 jbd2: Precompute number of transaction descriptor blocks

In reply to Jan Kara

Instead of computing the number of descriptor blocks a transaction can have each time we need it (which is currently when starting each transaction but will become more frequent later) precompute the number once during journal initialization together with maximum transaction size. We perform the precomputation whenever journal feature set is updated similarly as for computation of journal->j_revoke_records_per_block.

---
 fs/jbd2/journal.c     | 61 ++++++++++++++++++++++++++++++++-----------
 fs/jbd2/transaction.c | 24 +----------------
 include/linux/jbd2.h  |  7 +++++
 3 files changed, 54 insertions(+), 38 deletions(-)
fs/jbd2/journal.c +46 -15
index 1bb73750d307..ae5b544ed0cc 100644
@@ -1451,6 +1451,48 @@ static int journal_revoke_records_per_block(journal_t *journal)
 	return space / record_size;
 }
 
+static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
+{
+	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
+}
+
+/*
+ * Base amount of descriptor blocks we reserve for each transaction.
+ */
+static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
+{
+	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
+	int tags_per_block;
+
+	/* Subtract UUID */
+	tag_space -= 16;
+	if (jbd2_journal_has_csum_v2or3(journal))
+		tag_space -= sizeof(struct jbd2_journal_block_tail);
+	/* Commit code leaves a slack space of 16 bytes at the end of block */
+	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
+	/*
+	 * Revoke descriptors are accounted separately so we need to reserve
+	 * space for commit block and normal transaction descriptor blocks.
+	 */
+	return 1 + DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal),
+				tags_per_block);
+}
+
+/*
+ * Initialize number of blocks each transaction reserves for its bookkeeping
+ * and maximum number of blocks a transaction can use. This needs to be called
+ * after the journal size and the fastcommit area size are initialized.
+ */
+static void jbd2_journal_init_transaction_limits(journal_t *journal)
+{
+	journal->j_revoke_records_per_block =
+				journal_revoke_records_per_block(journal);
+	journal->j_transaction_overhead_buffers =
+				jbd2_descriptor_blocks_per_trans(journal);
+	journal->j_max_transaction_buffers =
+				jbd2_journal_get_max_txn_bufs(journal);
+}
+
 /*
  * Load the on-disk journal superblock and read the key fields into the
  * journal_t.
@@ -1492,8 +1534,8 @@ static int journal_load_superblock(journal_t *journal)
 	if (jbd2_journal_has_csum_v2or3(journal))
 		journal->j_csum_seed = jbd2_chksum(journal, ~0, sb->s_uuid,
 						   sizeof(sb->s_uuid));
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	/* After journal features are set, we can compute transaction limits */
+	jbd2_journal_init_transaction_limits(journal);
 
 	if (jbd2_has_feature_fast_commit(journal)) {
 		journal->j_fc_last = be32_to_cpu(sb->s_maxlen);
@@ -1698,11 +1740,6 @@ journal_t *jbd2_journal_init_inode(struct inode *inode)
 	return journal;
 }
 
-static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
-{
-	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
-}
-
 /*
  * Given a journal_t structure, initialise the various fields for
  * startup of a new journaling session.  We use this both when creating
@@ -1748,8 +1785,6 @@ static int journal_reset(journal_t *journal)
 	journal->j_commit_sequence = journal->j_transaction_sequence - 1;
 	journal->j_commit_request = journal->j_commit_sequence;
 
-	journal->j_max_transaction_buffers = jbd2_journal_get_max_txn_bufs(journal);
-
 	/*
 	 * Now that journal recovery is done, turn fast commits off here. This
 	 * way, if fast commit was enabled before the crash but if now FS has
@@ -2290,8 +2325,6 @@ jbd2_journal_initialize_fast_commit(journal_t *journal)
 	journal->j_fc_first = journal->j_last + 1;
 	journal->j_fc_off = 0;
 	journal->j_free = journal->j_last - journal->j_first;
-	journal->j_max_transaction_buffers =
-		jbd2_journal_get_max_txn_bufs(journal);
 
 	return 0;
 }
@@ -2379,8 +2412,7 @@ int jbd2_journal_set_features(journal_t *journal, unsigned long compat,
 	sb->s_feature_ro_compat |= cpu_to_be32(ro);
 	sb->s_feature_incompat  |= cpu_to_be32(incompat);
 	unlock_buffer(journal->j_sb_buffer);
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	jbd2_journal_init_transaction_limits(journal);
 
 	return 1;
 #undef COMPAT_FEATURE_ON
@@ -2411,8 +2443,7 @@ void jbd2_journal_clear_features(journal_t *journal, unsigned long compat,
 	sb->s_feature_compat    &= ~cpu_to_be32(compat);
 	sb->s_feature_ro_compat &= ~cpu_to_be32(ro);
 	sb->s_feature_incompat  &= ~cpu_to_be32(incompat);
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	jbd2_journal_init_transaction_limits(journal);
 }
 EXPORT_SYMBOL(jbd2_journal_clear_features);
 
fs/jbd2/transaction.c +1 -23
index cb0b8d6fc0c6..a095f1a3114b 100644
@@ -62,28 +62,6 @@ void jbd2_journal_free_transaction(transaction_t *transaction)
 	kmem_cache_free(transaction_cache, transaction);
 }
 
-/*
- * Base amount of descriptor blocks we reserve for each transaction.
- */
-static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
-{
-	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
-	int tags_per_block;
-
-	/* Subtract UUID */
-	tag_space -= 16;
-	if (jbd2_journal_has_csum_v2or3(journal))
-		tag_space -= sizeof(struct jbd2_journal_block_tail);
-	/* Commit code leaves a slack space of 16 bytes at the end of block */
-	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
-	/*
-	 * Revoke descriptors are accounted separately so we need to reserve
-	 * space for commit block and normal transaction descriptor blocks.
-	 */
-	return 1 + DIV_ROUND_UP(journal->j_max_transaction_buffers,
-				tags_per_block);
-}
-
 /*
  * jbd2_get_transaction: obtain a new transaction_t object.
  *
@@ -109,7 +87,7 @@ static void jbd2_get_transaction(journal_t *journal,
 	transaction->t_expires = jiffies + journal->j_commit_interval;
 	atomic_set(&transaction->t_updates, 0);
 	atomic_set(&transaction->t_outstanding_credits,
-		   jbd2_descriptor_blocks_per_trans(journal) +
+		   journal->j_transaction_overhead_buffers +
 		   atomic_read(&journal->j_reserved_credits));
 	atomic_set(&transaction->t_outstanding_revokes, 0);
 	atomic_set(&transaction->t_handle_count, 0);
include/linux/jbd2.h +7 -0
index f91b930abe20..b900c642210c 100644
@@ -1085,6 +1085,13 @@ struct journal_s
 	 */
 	int			j_revoke_records_per_block;
 
+	/**
+	 * @j_transaction_overhead:
+	 *
+	 * Number of blocks each transaction needs for its own bookkeeping
+	 */
+	int			j_transaction_overhead_buffers;
+
 	/**
 	 * @j_commit_interval:
 	 *
Signature · 1 line
2.35.3
#5 lore raw

Re: [PATCH v2 2/4] jbd2: Precompute number of transaction descriptor blocks

Hello Jan,

Quoted 40 lines · Jan Kara

Instead of computing the number of descriptor blocks a transaction can have each time we need it (which is currently when starting each transaction but will become more frequent later) precompute the number once during journal initialization together with maximum transaction size. We perform the precomputation whenever journal feature set is updated similarly as for computation of journal->j_revoke_records_per_block.

---
 fs/jbd2/journal.c     | 61 ++++++++++++++++++++++++++++++++-----------
 fs/jbd2/transaction.c | 24 +----------------
 include/linux/jbd2.h  |  7 +++++
 3 files changed, 54 insertions(+), 38 deletions(-)
fs/jbd2/journal.c +16 -0
index 1bb73750d307..ae5b544ed0cc 100644
@@ -1451,6 +1451,48 @@ static int journal_revoke_records_per_block(journal_t *journal)
 	return space / record_size;
 }
 
+static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
+{
+	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
+}
+
+/*
+ * Base amount of descriptor blocks we reserve for each transaction.
+ */
+static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
+{
+	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
+	int tags_per_block;
+
+	/* Subtract UUID */
+	tag_space -= 16;
+	if (jbd2_journal_has_csum_v2or3(journal))

fs/jbd2/journal.c static int journal_revoke_records_per_block(journal_t *journal) · line 1479

Quoted 10 lines
fs/jbd2/journal.c +10 -0
+		tag_space -= sizeof(struct jbd2_journal_block_tail);
+	/* Commit code leaves a slack space of 16 bytes at the end of block */
+	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
+	/*
+	 * Revoke descriptors are accounted separately so we need to reserve
+	 * space for commit block and normal transaction descriptor blocks.
+	 */
+	return 1 + DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal),
+				tags_per_block);
+}

The change looks good to me. I wonder if the original calculation of number of JBD2_DESCRIPTOR_BLOCK blocks is correct.
In my opinion, it should be:
DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal), tags_per_block *+ 1*) Assume max_txn_bufs is 6, tags_per_block is 1, then we have one tag block after each JBD2_DESCRIPTOR_BLOCK block. Then we could get 3 JBD2_DESCRIPTOR_BLOCK block at most rather than 6.
Please let me konw if I miss something, this confused me for sometime... Thanks.

Quoted 141 lines
fs/jbd2/journal.c +20 -15
+
+/*
+ * Initialize number of blocks each transaction reserves for its bookkeeping
+ * and maximum number of blocks a transaction can use. This needs to be called
+ * after the journal size and the fastcommit area size are initialized.
+ */
+static void jbd2_journal_init_transaction_limits(journal_t *journal)
+{
+	journal->j_revoke_records_per_block =
+				journal_revoke_records_per_block(journal);
+	journal->j_transaction_overhead_buffers =
+				jbd2_descriptor_blocks_per_trans(journal);
+	journal->j_max_transaction_buffers =
+				jbd2_journal_get_max_txn_bufs(journal);
+}
+
 /*
  * Load the on-disk journal superblock and read the key fields into the
  * journal_t.
@@ -1492,8 +1534,8 @@ static int journal_load_superblock(journal_t *journal)
 	if (jbd2_journal_has_csum_v2or3(journal))
 		journal->j_csum_seed = jbd2_chksum(journal, ~0, sb->s_uuid,
 						   sizeof(sb->s_uuid));
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	/* After journal features are set, we can compute transaction limits */
+	jbd2_journal_init_transaction_limits(journal);
 
 	if (jbd2_has_feature_fast_commit(journal)) {
 		journal->j_fc_last = be32_to_cpu(sb->s_maxlen);
@@ -1698,11 +1740,6 @@ journal_t *jbd2_journal_init_inode(struct inode *inode)
 	return journal;
 }
 
-static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
-{
-	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
-}
-
 /*
  * Given a journal_t structure, initialise the various fields for
  * startup of a new journaling session.  We use this both when creating
@@ -1748,8 +1785,6 @@ static int journal_reset(journal_t *journal)
 	journal->j_commit_sequence = journal->j_transaction_sequence - 1;
 	journal->j_commit_request = journal->j_commit_sequence;
 
-	journal->j_max_transaction_buffers = jbd2_journal_get_max_txn_bufs(journal);
-
 	/*
 	 * Now that journal recovery is done, turn fast commits off here. This
 	 * way, if fast commit was enabled before the crash but if now FS has
@@ -2290,8 +2325,6 @@ jbd2_journal_initialize_fast_commit(journal_t *journal)
 	journal->j_fc_first = journal->j_last + 1;
 	journal->j_fc_off = 0;
 	journal->j_free = journal->j_last - journal->j_first;
-	journal->j_max_transaction_buffers =
-		jbd2_journal_get_max_txn_bufs(journal);
 
 	return 0;
 }
@@ -2379,8 +2412,7 @@ int jbd2_journal_set_features(journal_t *journal, unsigned long compat,
 	sb->s_feature_ro_compat |= cpu_to_be32(ro);
 	sb->s_feature_incompat  |= cpu_to_be32(incompat);
 	unlock_buffer(journal->j_sb_buffer);
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	jbd2_journal_init_transaction_limits(journal);
 
 	return 1;
 #undef COMPAT_FEATURE_ON
@@ -2411,8 +2443,7 @@ void jbd2_journal_clear_features(journal_t *journal, unsigned long compat,
 	sb->s_feature_compat    &= ~cpu_to_be32(compat);
 	sb->s_feature_ro_compat &= ~cpu_to_be32(ro);
 	sb->s_feature_incompat  &= ~cpu_to_be32(incompat);
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	jbd2_journal_init_transaction_limits(journal);
 }
 EXPORT_SYMBOL(jbd2_journal_clear_features);
 
fs/jbd2/transaction.c +1 -23
index cb0b8d6fc0c6..a095f1a3114b 100644
@@ -62,28 +62,6 @@ void jbd2_journal_free_transaction(transaction_t *transaction)
 	kmem_cache_free(transaction_cache, transaction);
 }
 
-/*
- * Base amount of descriptor blocks we reserve for each transaction.
- */
-static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
-{
-	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
-	int tags_per_block;
-
-	/* Subtract UUID */
-	tag_space -= 16;
-	if (jbd2_journal_has_csum_v2or3(journal))
-		tag_space -= sizeof(struct jbd2_journal_block_tail);
-	/* Commit code leaves a slack space of 16 bytes at the end of block */
-	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
-	/*
-	 * Revoke descriptors are accounted separately so we need to reserve
-	 * space for commit block and normal transaction descriptor blocks.
-	 */
-	return 1 + DIV_ROUND_UP(journal->j_max_transaction_buffers,
-				tags_per_block);
-}
-
 /*
  * jbd2_get_transaction: obtain a new transaction_t object.
  *
@@ -109,7 +87,7 @@ static void jbd2_get_transaction(journal_t *journal,
 	transaction->t_expires = jiffies + journal->j_commit_interval;
 	atomic_set(&transaction->t_updates, 0);
 	atomic_set(&transaction->t_outstanding_credits,
-		   jbd2_descriptor_blocks_per_trans(journal) +
+		   journal->j_transaction_overhead_buffers +
 		   atomic_read(&journal->j_reserved_credits));
 	atomic_set(&transaction->t_outstanding_revokes, 0);
 	atomic_set(&transaction->t_handle_count, 0);
include/linux/jbd2.h +7 -0
index f91b930abe20..b900c642210c 100644
@@ -1085,6 +1085,13 @@ struct journal_s
 	 */
 	int			j_revoke_records_per_block;
 
+	/**
+	 * @j_transaction_overhead:
+	 *
+	 * Number of blocks each transaction needs for its own bookkeeping
+	 */
+	int			j_transaction_overhead_buffers;
+
 	/**
 	 * @j_commit_interval:
 	 *
author
#6 lore raw

Re: [PATCH v2 2/4] jbd2: Precompute number of transaction descriptor blocks

Quoted 35 lines · Kemeng Shi
Quoted 24 lines · Jan Kara

Instead of computing the number of descriptor blocks a transaction can have each time we need it (which is currently when starting each transaction but will become more frequent later) precompute the number once during journal initialization together with maximum transaction size. We perform the precomputation whenever journal feature set is updated similarly as for computation of journal->j_revoke_records_per_block.

---
 fs/jbd2/journal.c     | 61 ++++++++++++++++++++++++++++++++-----------
 fs/jbd2/transaction.c | 24 +----------------
 include/linux/jbd2.h  |  7 +++++
 3 files changed, 54 insertions(+), 38 deletions(-)
fs/jbd2/journal.c +8 -0
+static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
+{
+	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
+	int tags_per_block;
+
+	/* Subtract UUID */
+	tag_space -= 16;
+	if (jbd2_journal_has_csum_v2or3(journal))

fs/jbd2/journal.c static int journal_revoke_records_per_block(journal_t *journal) · line 1479

Quoted 10 lines
fs/jbd2/journal.c +10 -0
+		tag_space -= sizeof(struct jbd2_journal_block_tail);
+	/* Commit code leaves a slack space of 16 bytes at the end of block */
+	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
+	/*
+	 * Revoke descriptors are accounted separately so we need to reserve
+	 * space for commit block and normal transaction descriptor blocks.
+	 */
+	return 1 + DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal),
+				tags_per_block);
+}
Quoted 8 lines

The change looks good to me. I wonder if the original calculation of number of JBD2_DESCRIPTOR_BLOCK blocks is correct.
In my opinion, it should be:
DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal), tags_per_block *+ 1*) Assume max_txn_bufs is 6, tags_per_block is 1, then we have one tag block after each JBD2_DESCRIPTOR_BLOCK block. Then we could get 3 JBD2_DESCRIPTOR_BLOCK block at most rather than 6.
Please let me konw if I miss something, this confused me for sometime...

So you are correct that the expression is overestimating the number of descriptor blocks required, essentially because we don't need descriptor blocks for descriptor blocks. But given tags_per_block is at least over 60, in common configurations over 250, this imprecision is not really substantial and I prefer a simple to argue about upper estimate...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#7 lore raw

Re: [PATCH v2 2/4] jbd2: Precompute number of transaction descriptor blocks

Quoted 45 lines · Jan Kara
Quoted 35 lines · Kemeng Shi
Quoted 24 lines · Jan Kara

Instead of computing the number of descriptor blocks a transaction can have each time we need it (which is currently when starting each transaction but will become more frequent later) precompute the number once during journal initialization together with maximum transaction size. We perform the precomputation whenever journal feature set is updated similarly as for computation of journal->j_revoke_records_per_block.

---
 fs/jbd2/journal.c     | 61 ++++++++++++++++++++++++++++++++-----------
 fs/jbd2/transaction.c | 24 +----------------
 include/linux/jbd2.h  |  7 +++++
 3 files changed, 54 insertions(+), 38 deletions(-)
fs/jbd2/journal.c +8 -0
+static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
+{
+	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
+	int tags_per_block;
+
+	/* Subtract UUID */
+	tag_space -= 16;
+	if (jbd2_journal_has_csum_v2or3(journal))

fs/jbd2/journal.c static int journal_revoke_records_per_block(journal_t *journal) · line 1479

Quoted 10 lines
fs/jbd2/journal.c +10 -0
+		tag_space -= sizeof(struct jbd2_journal_block_tail);
+	/* Commit code leaves a slack space of 16 bytes at the end of block */
+	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
+	/*
+	 * Revoke descriptors are accounted separately so we need to reserve
+	 * space for commit block and normal transaction descriptor blocks.
+	 */
+	return 1 + DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal),
+				tags_per_block);
+}
Quoted 8 lines

The change looks good to me. I wonder if the original calculation of number of JBD2_DESCRIPTOR_BLOCK blocks is correct.
In my opinion, it should be:
DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal), tags_per_block *+ 1*) Assume max_txn_bufs is 6, tags_per_block is 1, then we have one tag block after each JBD2_DESCRIPTOR_BLOCK block. Then we could get 3 JBD2_DESCRIPTOR_BLOCK block at most rather than 6.
Please let me konw if I miss something, this confused me for sometime...

Quoted 5 lines

So you are correct that the expression is overestimating the number of descriptor blocks required, essentially because we don't need descriptor blocks for descriptor blocks. But given tags_per_block is at least over 60, in common configurations over 250, this imprecision is not really substantial and I prefer a simple to argue about upper estimate...

Right, I agree with this. Thanks for explanation.

Quoted 3 lines

Honza

Reviewed-by on v2 2/4
#8 lore raw

Re: [PATCH v2 2/4] jbd2: Precompute number of transaction descriptor blocks

In reply to Jan Kara
Quoted 10 lines · Jan Kara

Instead of computing the number of descriptor blocks a transaction can have each time we need it (which is currently when starting each transaction but will become more frequent later) precompute the number once during journal initialization together with maximum transaction size. We perform the precomputation whenever journal feature set is updated similarly as for computation of journal->j_revoke_records_per_block.

Looks good to me.

Quoted 181 lines
---
 fs/jbd2/journal.c     | 61 ++++++++++++++++++++++++++++++++-----------
 fs/jbd2/transaction.c | 24 +----------------
 include/linux/jbd2.h  |  7 +++++
 3 files changed, 54 insertions(+), 38 deletions(-)
fs/jbd2/journal.c +46 -15
index 1bb73750d307..ae5b544ed0cc 100644
@@ -1451,6 +1451,48 @@ static int journal_revoke_records_per_block(journal_t *journal)
 	return space / record_size;
 }
 
+static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
+{
+	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
+}
+
+/*
+ * Base amount of descriptor blocks we reserve for each transaction.
+ */
+static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
+{
+	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
+	int tags_per_block;
+
+	/* Subtract UUID */
+	tag_space -= 16;
+	if (jbd2_journal_has_csum_v2or3(journal))
+		tag_space -= sizeof(struct jbd2_journal_block_tail);
+	/* Commit code leaves a slack space of 16 bytes at the end of block */
+	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
+	/*
+	 * Revoke descriptors are accounted separately so we need to reserve
+	 * space for commit block and normal transaction descriptor blocks.
+	 */
+	return 1 + DIV_ROUND_UP(jbd2_journal_get_max_txn_bufs(journal),
+				tags_per_block);
+}
+
+/*
+ * Initialize number of blocks each transaction reserves for its bookkeeping
+ * and maximum number of blocks a transaction can use. This needs to be called
+ * after the journal size and the fastcommit area size are initialized.
+ */
+static void jbd2_journal_init_transaction_limits(journal_t *journal)
+{
+	journal->j_revoke_records_per_block =
+				journal_revoke_records_per_block(journal);
+	journal->j_transaction_overhead_buffers =
+				jbd2_descriptor_blocks_per_trans(journal);
+	journal->j_max_transaction_buffers =
+				jbd2_journal_get_max_txn_bufs(journal);
+}
+
 /*
  * Load the on-disk journal superblock and read the key fields into the
  * journal_t.
@@ -1492,8 +1534,8 @@ static int journal_load_superblock(journal_t *journal)
 	if (jbd2_journal_has_csum_v2or3(journal))
 		journal->j_csum_seed = jbd2_chksum(journal, ~0, sb->s_uuid,
 						   sizeof(sb->s_uuid));
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	/* After journal features are set, we can compute transaction limits */
+	jbd2_journal_init_transaction_limits(journal);
 
 	if (jbd2_has_feature_fast_commit(journal)) {
 		journal->j_fc_last = be32_to_cpu(sb->s_maxlen);
@@ -1698,11 +1740,6 @@ journal_t *jbd2_journal_init_inode(struct inode *inode)
 	return journal;
 }
 
-static int jbd2_journal_get_max_txn_bufs(journal_t *journal)
-{
-	return (journal->j_total_len - journal->j_fc_wbufsize) / 4;
-}
-
 /*
  * Given a journal_t structure, initialise the various fields for
  * startup of a new journaling session.  We use this both when creating
@@ -1748,8 +1785,6 @@ static int journal_reset(journal_t *journal)
 	journal->j_commit_sequence = journal->j_transaction_sequence - 1;
 	journal->j_commit_request = journal->j_commit_sequence;
 
-	journal->j_max_transaction_buffers = jbd2_journal_get_max_txn_bufs(journal);
-
 	/*
 	 * Now that journal recovery is done, turn fast commits off here. This
 	 * way, if fast commit was enabled before the crash but if now FS has
@@ -2290,8 +2325,6 @@ jbd2_journal_initialize_fast_commit(journal_t *journal)
 	journal->j_fc_first = journal->j_last + 1;
 	journal->j_fc_off = 0;
 	journal->j_free = journal->j_last - journal->j_first;
-	journal->j_max_transaction_buffers =
-		jbd2_journal_get_max_txn_bufs(journal);
 
 	return 0;
 }
@@ -2379,8 +2412,7 @@ int jbd2_journal_set_features(journal_t *journal, unsigned long compat,
 	sb->s_feature_ro_compat |= cpu_to_be32(ro);
 	sb->s_feature_incompat  |= cpu_to_be32(incompat);
 	unlock_buffer(journal->j_sb_buffer);
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	jbd2_journal_init_transaction_limits(journal);
 
 	return 1;
 #undef COMPAT_FEATURE_ON
@@ -2411,8 +2443,7 @@ void jbd2_journal_clear_features(journal_t *journal, unsigned long compat,
 	sb->s_feature_compat    &= ~cpu_to_be32(compat);
 	sb->s_feature_ro_compat &= ~cpu_to_be32(ro);
 	sb->s_feature_incompat  &= ~cpu_to_be32(incompat);
-	journal->j_revoke_records_per_block =
-				journal_revoke_records_per_block(journal);
+	jbd2_journal_init_transaction_limits(journal);
 }
 EXPORT_SYMBOL(jbd2_journal_clear_features);
 
fs/jbd2/transaction.c +1 -23
index cb0b8d6fc0c6..a095f1a3114b 100644
@@ -62,28 +62,6 @@ void jbd2_journal_free_transaction(transaction_t *transaction)
 	kmem_cache_free(transaction_cache, transaction);
 }
 
-/*
- * Base amount of descriptor blocks we reserve for each transaction.
- */
-static int jbd2_descriptor_blocks_per_trans(journal_t *journal)
-{
-	int tag_space = journal->j_blocksize - sizeof(journal_header_t);
-	int tags_per_block;
-
-	/* Subtract UUID */
-	tag_space -= 16;
-	if (jbd2_journal_has_csum_v2or3(journal))
-		tag_space -= sizeof(struct jbd2_journal_block_tail);
-	/* Commit code leaves a slack space of 16 bytes at the end of block */
-	tags_per_block = (tag_space - 16) / journal_tag_bytes(journal);
-	/*
-	 * Revoke descriptors are accounted separately so we need to reserve
-	 * space for commit block and normal transaction descriptor blocks.
-	 */
-	return 1 + DIV_ROUND_UP(journal->j_max_transaction_buffers,
-				tags_per_block);
-}
-
 /*
  * jbd2_get_transaction: obtain a new transaction_t object.
  *
@@ -109,7 +87,7 @@ static void jbd2_get_transaction(journal_t *journal,
 	transaction->t_expires = jiffies + journal->j_commit_interval;
 	atomic_set(&transaction->t_updates, 0);
 	atomic_set(&transaction->t_outstanding_credits,
-		   jbd2_descriptor_blocks_per_trans(journal) +
+		   journal->j_transaction_overhead_buffers +
 		   atomic_read(&journal->j_reserved_credits));
 	atomic_set(&transaction->t_outstanding_revokes, 0);
 	atomic_set(&transaction->t_handle_count, 0);
include/linux/jbd2.h +7 -0
index f91b930abe20..b900c642210c 100644
@@ -1085,6 +1085,13 @@ struct journal_s
 	 */
 	int			j_revoke_records_per_block;
 
+	/**
+	 * @j_transaction_overhead:
+	 *
+	 * Number of blocks each transaction needs for its own bookkeeping
+	 */
+	int			j_transaction_overhead_buffers;
+
 	/**
 	 * @j_commit_interval:
 	 *
author
#9 lore raw

PATCH v2 3/4 jbd2: Avoid infinite transaction commit loop

In reply to Jan Kara

Commit 9f356e5a4f12 ("jbd2: Account descriptor blocks into t_outstanding_credits") started to account descriptor blocks into transactions outstanding credits. However it didn't appropriately decrease the maximum amount of credits available to userspace. Thus if the filesystem requests a transaction smaller than j_max_transaction_buffers but large enough that when descriptor blocks are added the size exceeds j_max_transaction_buffers, we confuse add_transaction_credits() into thinking previous handles have grown the transaction too much and enter infinite journal commit loop in start_this_handle() -> add_transaction_credits() trying to create transaction with enough credits available.

Fix the problem by properly accounting for transaction space reserved for descriptor blocks when verifying requested transaction handle size.

---
 fs/jbd2/transaction.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
fs/jbd2/transaction.c +14 -7
index a095f1a3114b..66513c18ca29 100644
@@ -191,6 +191,13 @@ static void sub_reserved_credits(journal_t *journal, int blocks)
 	wake_up(&journal->j_wait_reserved);
 }
 
+/* Maximum number of blocks for user transaction payload */
+static int jbd2_max_user_trans_buffers(journal_t *journal)
+{
+	return journal->j_max_transaction_buffers -
+				journal->j_transaction_overhead_buffers;
+}
+
 /*
  * Wait until we can add credits for handle to the running transaction.  Called
  * with j_state_lock held for reading. Returns 0 if handle joined the running
@@ -240,12 +247,12 @@ __must_hold(&journal->j_state_lock)
 		 * big to fit this handle? Wait until reserved credits are freed.
 		 */
 		if (atomic_read(&journal->j_reserved_credits) + total >
-		    journal->j_max_transaction_buffers) {
+		    jbd2_max_user_trans_buffers(journal)) {
 			read_unlock(&journal->j_state_lock);
 			jbd2_might_wait_for_commit(journal);
 			wait_event(journal->j_wait_reserved,
 				   atomic_read(&journal->j_reserved_credits) + total <=
-				   journal->j_max_transaction_buffers);
+				   jbd2_max_user_trans_buffers(journal));
 			__acquire(&journal->j_state_lock); /* fake out sparse */
 			return 1;
 		}
@@ -285,14 +292,14 @@ __must_hold(&journal->j_state_lock)
 
 	needed = atomic_add_return(rsv_blocks, &journal->j_reserved_credits);
 	/* We allow at most half of a transaction to be reserved */
-	if (needed > journal->j_max_transaction_buffers / 2) {
+	if (needed > jbd2_max_user_trans_buffers(journal) / 2) {
 		sub_reserved_credits(journal, rsv_blocks);
 		atomic_sub(total, &t->t_outstanding_credits);
 		read_unlock(&journal->j_state_lock);
 		jbd2_might_wait_for_commit(journal);
 		wait_event(journal->j_wait_reserved,
 			 atomic_read(&journal->j_reserved_credits) + rsv_blocks
-			 <= journal->j_max_transaction_buffers / 2);
+			 <= jbd2_max_user_trans_buffers(journal) / 2);
 		__acquire(&journal->j_state_lock); /* fake out sparse */
 		return 1;
 	}
@@ -322,12 +329,12 @@ static int start_this_handle(journal_t *journal, handle_t *handle,
 	 * size and limit the number of total credits to not exceed maximum
 	 * transaction size per operation.
 	 */
-	if ((rsv_blocks > journal->j_max_transaction_buffers / 2) ||
-	    (rsv_blocks + blocks > journal->j_max_transaction_buffers)) {
+	if (rsv_blocks > jbd2_max_user_trans_buffers(journal) / 2 ||
+	    rsv_blocks + blocks > jbd2_max_user_trans_buffers(journal)) {
 		printk(KERN_ERR "JBD2: %s wants too many credits "
 		       "credits:%d rsv_credits:%d max:%d\n",
 		       current->comm, blocks, rsv_blocks,
-		       journal->j_max_transaction_buffers);
+		       jbd2_max_user_trans_buffers(journal));
 		WARN_ON(1);
 		return -ENOSPC;
 	}
Signature · 1 line
2.35.3
#10 lore raw
Quoted 12 lines · Jan Kara

Commit 9f356e5a4f12 ("jbd2: Account descriptor blocks into t_outstanding_credits") started to account descriptor blocks into transactions outstanding credits. However it didn't appropriately decrease the maximum amount of credits available to userspace. Thus if the filesystem requests a transaction smaller than j_max_transaction_buffers but large enough that when descriptor blocks are added the size exceeds j_max_transaction_buffers, we confuse add_transaction_credits() into thinking previous handles have grown the transaction too much and enter infinite journal commit loop in start_this_handle() -> add_transaction_credits() trying to create transaction with enough credits available.

Hello Jan!

I understand that the incorrect max transaction limit in start_this_handle() could lead to infinite loop in start_this_handle()-> add_transaction_credits() with large enough userspace credits (from j_max_transaction_buffers - overheads to j_max_transaction_buffers), but I don't get how could it lead to ran out of space in the journal commit traction? IIUC, below codes in add_transaction_credits() could make sure that we have enough space when committing traction:

static int add_transaction_credits()
{
...

	if (jbd2_log_space_left(journal) < journal->j_max_transaction_buffers) {
		...
		return 1;
		...
	}

...
}

I can't open and download the image Alexander gave, so I can't get to the bottom of this issue, please let me know what happened with jbd2_journal_next_log_block().

Thanks,
Yi.

Quoted 79 lines

Fix the problem by properly accounting for transaction space reserved for descriptor blocks when verifying requested transaction handle size.

---
 fs/jbd2/transaction.c | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)
fs/jbd2/transaction.c +14 -7
index a095f1a3114b..66513c18ca29 100644
@@ -191,6 +191,13 @@ static void sub_reserved_credits(journal_t *journal, int blocks)
 	wake_up(&journal->j_wait_reserved);
 }
 
+/* Maximum number of blocks for user transaction payload */
+static int jbd2_max_user_trans_buffers(journal_t *journal)
+{
+	return journal->j_max_transaction_buffers -
+				journal->j_transaction_overhead_buffers;
+}
+
 /*
  * Wait until we can add credits for handle to the running transaction.  Called
  * with j_state_lock held for reading. Returns 0 if handle joined the running
@@ -240,12 +247,12 @@ __must_hold(&journal->j_state_lock)
 		 * big to fit this handle? Wait until reserved credits are freed.
 		 */
 		if (atomic_read(&journal->j_reserved_credits) + total >
-		    journal->j_max_transaction_buffers) {
+		    jbd2_max_user_trans_buffers(journal)) {
 			read_unlock(&journal->j_state_lock);
 			jbd2_might_wait_for_commit(journal);
 			wait_event(journal->j_wait_reserved,
 				   atomic_read(&journal->j_reserved_credits) + total <=
-				   journal->j_max_transaction_buffers);
+				   jbd2_max_user_trans_buffers(journal));
 			__acquire(&journal->j_state_lock); /* fake out sparse */
 			return 1;
 		}
@@ -285,14 +292,14 @@ __must_hold(&journal->j_state_lock)
 
 	needed = atomic_add_return(rsv_blocks, &journal->j_reserved_credits);
 	/* We allow at most half of a transaction to be reserved */
-	if (needed > journal->j_max_transaction_buffers / 2) {
+	if (needed > jbd2_max_user_trans_buffers(journal) / 2) {
 		sub_reserved_credits(journal, rsv_blocks);
 		atomic_sub(total, &t->t_outstanding_credits);
 		read_unlock(&journal->j_state_lock);
 		jbd2_might_wait_for_commit(journal);
 		wait_event(journal->j_wait_reserved,
 			 atomic_read(&journal->j_reserved_credits) + rsv_blocks
-			 <= journal->j_max_transaction_buffers / 2);
+			 <= jbd2_max_user_trans_buffers(journal) / 2);
 		__acquire(&journal->j_state_lock); /* fake out sparse */
 		return 1;
 	}
@@ -322,12 +329,12 @@ static int start_this_handle(journal_t *journal, handle_t *handle,
 	 * size and limit the number of total credits to not exceed maximum
 	 * transaction size per operation.
 	 */
-	if ((rsv_blocks > journal->j_max_transaction_buffers / 2) ||
-	    (rsv_blocks + blocks > journal->j_max_transaction_buffers)) {
+	if (rsv_blocks > jbd2_max_user_trans_buffers(journal) / 2 ||
+	    rsv_blocks + blocks > jbd2_max_user_trans_buffers(journal)) {
 		printk(KERN_ERR "JBD2: %s wants too many credits "
 		       "credits:%d rsv_credits:%d max:%d\n",
 		       current->comm, blocks, rsv_blocks,
-		       journal->j_max_transaction_buffers);
+		       jbd2_max_user_trans_buffers(journal));
 		WARN_ON(1);
 		return -ENOSPC;
 	}
author
#11 lore raw
Quoted 30 lines · Zhang Yi
Quoted 11 lines · Jan Kara

Commit 9f356e5a4f12 ("jbd2: Account descriptor blocks into t_outstanding_credits") started to account descriptor blocks into transactions outstanding credits. However it didn't appropriately decrease the maximum amount of credits available to userspace. Thus if the filesystem requests a transaction smaller than j_max_transaction_buffers but large enough that when descriptor blocks are added the size exceeds j_max_transaction_buffers, we confuse add_transaction_credits() into thinking previous handles have grown the transaction too much and enter infinite journal commit loop in start_this_handle() -> add_transaction_credits() trying to create transaction with enough credits available.

I understand that the incorrect max transaction limit in start_this_handle() could lead to infinite loop in start_this_handle()-> add_transaction_credits() with large enough userspace credits (from j_max_transaction_buffers - overheads to j_max_transaction_buffers), but I don't get how could it lead to ran out of space in the journal commit traction? IIUC, below codes in add_transaction_credits() could make sure that we have enough space when committing traction:

static int add_transaction_credits()
{
...

	if (jbd2_log_space_left(journal) < journal->j_max_transaction_buffers) {
		...
		return 1;
		...
	}
Quoted 6 lines

...
}

I can't open and download the image Alexander gave, so I can't get to the bottom of this issue, please let me know what happened with jbd2_journal_next_log_block().

Sure. So what was exactly happening is a loop like this:

start_this_handle()
blocks = 252 (handle->h_total_credits)
- starts a new transaction
- t_outstanding_credits set to 6 to account for the commit block and descriptor blocks
add_transaction_credits(journal, 252, 0)

     needed = atomic_add_return(252, &t->t_outstanding_credits);
     if (needed > journal->j_max_transaction_buffers) {
	/* Yes, this is exceeded due to descriptor blocks being in
	 * t_outstanding_credits */
        ...
        wait_transaction_locked(journal);
	  - this commits an empty transaction - contains only the commit
	    block
        return 1

goto repeat

So we commit single block transactions in a loop until we exhaust all the journal space. The condition in add_transaction_credits() whether there's enough space in the journal is never reached so we don't ever push the journal tail to make space in the journal.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#12 lore raw
Quoted 55 lines · Jan Kara
Quoted 30 lines · Zhang Yi
Quoted 11 lines · Jan Kara

Commit 9f356e5a4f12 ("jbd2: Account descriptor blocks into t_outstanding_credits") started to account descriptor blocks into transactions outstanding credits. However it didn't appropriately decrease the maximum amount of credits available to userspace. Thus if the filesystem requests a transaction smaller than j_max_transaction_buffers but large enough that when descriptor blocks are added the size exceeds j_max_transaction_buffers, we confuse add_transaction_credits() into thinking previous handles have grown the transaction too much and enter infinite journal commit loop in start_this_handle() -> add_transaction_credits() trying to create transaction with enough credits available.

I understand that the incorrect max transaction limit in start_this_handle() could lead to infinite loop in start_this_handle()-> add_transaction_credits() with large enough userspace credits (from j_max_transaction_buffers - overheads to j_max_transaction_buffers), but I don't get how could it lead to ran out of space in the journal commit traction? IIUC, below codes in add_transaction_credits() could make sure that we have enough space when committing traction:

static int add_transaction_credits()
{
...

	if (jbd2_log_space_left(journal) < journal->j_max_transaction_buffers) {
		...
		return 1;
		...
	}
Quoted 6 lines

...
}

I can't open and download the image Alexander gave, so I can't get to the bottom of this issue, please let me know what happened with jbd2_journal_next_log_block().

Sure. So what was exactly happening is a loop like this:

start_this_handle()
blocks = 252 (handle->h_total_credits)
- starts a new transaction
- t_outstanding_credits set to 6 to account for the commit block and descriptor blocks
add_transaction_credits(journal, 252, 0)

     needed = atomic_add_return(252, &t->t_outstanding_credits);
     if (needed > journal->j_max_transaction_buffers) {
	/* Yes, this is exceeded due to descriptor blocks being in
	 * t_outstanding_credits */
        ...
        wait_transaction_locked(journal);
	  - this commits an empty transaction - contains only the commit
	    block
        return 1
Quoted 7 lines

goto repeat

So we commit single block transactions in a loop until we exhaust all the journal space. The condition in add_transaction_credits() whether there's enough space in the journal is never reached so we don't ever push the journal tail to make space in the journal.

mm-hm, ha, yeah, this will lead to submit an empty transaction in each loop, but I still have one question, although the journal tail can't be updated in add_transaction_credits(), I think the journal tail should be updated in jbd2_journal_commit_transaction()->jbd2_update_log_tail() since we don't add empty transactions to journal->j_checkpoint_transactions list, the loop in jbd2_journal_commit_transaction() should like this:

...
jbd2_journal_commit_transaction()
update_tail = jbd2_journal_get_log_tail()

    //journal->j_checkpoint_transactions should be NULL, tid is the
    //committing transaction's tid, which should be large than the
    //tail tid since the second loop, so this should be true after
    //the second loop
  if (freed < journal->j_max_transaction_buffers)
    update_tail = 0;
    //update_tail should be true after j_max_transaction_buffers loop

jbd2_update_log_tail() //j_free should be increased in each

                        //j_max_transaction_buffers loop
  if (tid_gt(tid, journal->j_tail_sequence)) //it's true
  __jbd2_update_log_tail()
    journal->j_free += freed; //update log tail and increase j_free
                              //j_max_transaction_buffers blocks

...

As I understand it, the journal space can't be exhausted, I don't know how it happened, am I missing something?

Thanks,
Yi.

author
#13 lore raw
Quoted 88 lines · Zhang Yi
Quoted 55 lines · Jan Kara
Quoted 30 lines · Zhang Yi
Quoted 11 lines · Jan Kara

Commit 9f356e5a4f12 ("jbd2: Account descriptor blocks into t_outstanding_credits") started to account descriptor blocks into transactions outstanding credits. However it didn't appropriately decrease the maximum amount of credits available to userspace. Thus if the filesystem requests a transaction smaller than j_max_transaction_buffers but large enough that when descriptor blocks are added the size exceeds j_max_transaction_buffers, we confuse add_transaction_credits() into thinking previous handles have grown the transaction too much and enter infinite journal commit loop in start_this_handle() -> add_transaction_credits() trying to create transaction with enough credits available.

I understand that the incorrect max transaction limit in start_this_handle() could lead to infinite loop in start_this_handle()-> add_transaction_credits() with large enough userspace credits (from j_max_transaction_buffers - overheads to j_max_transaction_buffers), but I don't get how could it lead to ran out of space in the journal commit traction? IIUC, below codes in add_transaction_credits() could make sure that we have enough space when committing traction:

static int add_transaction_credits()
{
...

	if (jbd2_log_space_left(journal) < journal->j_max_transaction_buffers) {
		...
		return 1;
		...
	}
Quoted 6 lines

...
}

I can't open and download the image Alexander gave, so I can't get to the bottom of this issue, please let me know what happened with jbd2_journal_next_log_block().

Sure. So what was exactly happening is a loop like this:

start_this_handle()
blocks = 252 (handle->h_total_credits)
- starts a new transaction
- t_outstanding_credits set to 6 to account for the commit block and descriptor blocks
add_transaction_credits(journal, 252, 0)

     needed = atomic_add_return(252, &t->t_outstanding_credits);
     if (needed > journal->j_max_transaction_buffers) {
	/* Yes, this is exceeded due to descriptor blocks being in
	 * t_outstanding_credits */
        ...
        wait_transaction_locked(journal);
	  - this commits an empty transaction - contains only the commit
	    block
        return 1
Quoted 7 lines

goto repeat

So we commit single block transactions in a loop until we exhaust all the journal space. The condition in add_transaction_credits() whether there's enough space in the journal is never reached so we don't ever push the journal tail to make space in the journal.

mm-hm, ha, yeah, this will lead to submit an empty transaction in each loop, but I still have one question, although the journal tail can't be updated in add_transaction_credits(), I think the journal tail should be updated in jbd2_journal_commit_transaction()->jbd2_update_log_tail() since we don't add empty transactions to journal->j_checkpoint_transactions list, the loop in jbd2_journal_commit_transaction() should like this:

...
jbd2_journal_commit_transaction()
update_tail = jbd2_journal_get_log_tail()

    //journal->j_checkpoint_transactions should be NULL, tid is the
    //committing transaction's tid, which should be large than the
    //tail tid since the second loop, so this should be true after
    //the second loop
  if (freed < journal->j_max_transaction_buffers)
    update_tail = 0;
    //update_tail should be true after j_max_transaction_buffers loop

jbd2_update_log_tail() //j_free should be increased in each

                        //j_max_transaction_buffers loop
  if (tid_gt(tid, journal->j_tail_sequence)) //it's true
  __jbd2_update_log_tail()
    journal->j_free += freed; //update log tail and increase j_free
                              //j_max_transaction_buffers blocks
Quoted 4 lines

...

As I understand it, the journal space can't be exhausted, I don't know how it happened, am I missing something?

Well, at the beginning of the journal there are a few non-empty transactions whose buffers didn't get checkpointed before we entered the commit loop. So we cannot just push the journal tail without doing a checkpoint and we never call into the checkpointing code in the commit loop...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Reviewed-by
#14 lore raw
Quoted 91 lines · Jan Kara
Quoted 90 lines · Zhang Yi
Quoted 55 lines · Jan Kara
Quoted 30 lines · Zhang Yi
Quoted 11 lines · Jan Kara

Commit 9f356e5a4f12 ("jbd2: Account descriptor blocks into t_outstanding_credits") started to account descriptor blocks into transactions outstanding credits. However it didn't appropriately decrease the maximum amount of credits available to userspace. Thus if the filesystem requests a transaction smaller than j_max_transaction_buffers but large enough that when descriptor blocks are added the size exceeds j_max_transaction_buffers, we confuse add_transaction_credits() into thinking previous handles have grown the transaction too much and enter infinite journal commit loop in start_this_handle() -> add_transaction_credits() trying to create transaction with enough credits available.

I understand that the incorrect max transaction limit in start_this_handle() could lead to infinite loop in start_this_handle()-> add_transaction_credits() with large enough userspace credits (from j_max_transaction_buffers - overheads to j_max_transaction_buffers), but I don't get how could it lead to ran out of space in the journal commit traction? IIUC, below codes in add_transaction_credits() could make sure that we have enough space when committing traction:

static int add_transaction_credits()
{
...

	if (jbd2_log_space_left(journal) < journal->j_max_transaction_buffers) {
		...
		return 1;
		...
	}
Quoted 6 lines

...
}

I can't open and download the image Alexander gave, so I can't get to the bottom of this issue, please let me know what happened with jbd2_journal_next_log_block().

Sure. So what was exactly happening is a loop like this:

start_this_handle()
blocks = 252 (handle->h_total_credits)
- starts a new transaction
- t_outstanding_credits set to 6 to account for the commit block and descriptor blocks
add_transaction_credits(journal, 252, 0)

     needed = atomic_add_return(252, &t->t_outstanding_credits);
     if (needed > journal->j_max_transaction_buffers) {
	/* Yes, this is exceeded due to descriptor blocks being in
	 * t_outstanding_credits */
        ...
        wait_transaction_locked(journal);
	  - this commits an empty transaction - contains only the commit
	    block
        return 1
Quoted 7 lines

goto repeat

So we commit single block transactions in a loop until we exhaust all the journal space. The condition in add_transaction_credits() whether there's enough space in the journal is never reached so we don't ever push the journal tail to make space in the journal.

mm-hm, ha, yeah, this will lead to submit an empty transaction in each loop, but I still have one question, although the journal tail can't be updated in add_transaction_credits(), I think the journal tail should be updated in jbd2_journal_commit_transaction()->jbd2_update_log_tail() since we don't add empty transactions to journal->j_checkpoint_transactions list, the loop in jbd2_journal_commit_transaction() should like this:

...
jbd2_journal_commit_transaction()
update_tail = jbd2_journal_get_log_tail()

    //journal->j_checkpoint_transactions should be NULL, tid is the
    //committing transaction's tid, which should be large than the
    //tail tid since the second loop, so this should be true after
    //the second loop
  if (freed < journal->j_max_transaction_buffers)
    update_tail = 0;
    //update_tail should be true after j_max_transaction_buffers loop

jbd2_update_log_tail() //j_free should be increased in each

                        //j_max_transaction_buffers loop
  if (tid_gt(tid, journal->j_tail_sequence)) //it's true
  __jbd2_update_log_tail()
    journal->j_free += freed; //update log tail and increase j_free
                              //j_max_transaction_buffers blocks

...

Quoted 8 lines
Quoted 2 lines

As I understand it, the journal space can't be exhausted, I don't know how it happened, am I missing something?

Well, at the beginning of the journal there are a few non-empty transactions whose buffers didn't get checkpointed before we entered the commit loop. So we cannot just push the journal tail without doing a checkpoint and we never call into the checkpointing code in the commit loop...

Oh, indeed, I see, thanks for explaining and this patch looks good to me.

Thanks,
Yi.

author
#15 lore raw

PATCH v2 4/4 jbd2: Drop pointless shrinker batch initialization

In reply to Jan Kara

In jbd2_journal_init_common() we set batch size of a shrinker shrinking checkpointed buffers to journal->j_max_transaction_buffers. But that is guaranteed to be 0 at that point so we effectively stay with the default shrinker batch size of 128. It has been like this since introduction of jbd2 shrinkers so just drop the pointless initialization.

---
 fs/jbd2/journal.c | 1 -
 1 file changed, 1 deletion(-)
fs/jbd2/journal.c +0 -1
index ae5b544ed0cc..c356cc027ed7 100644
@@ -1641,7 +1641,6 @@ static journal_t *journal_init_common(struct block_device *bdev,
 
 	journal->j_shrinker->scan_objects = jbd2_journal_shrink_scan;
 	journal->j_shrinker->count_objects = jbd2_journal_shrink_count;
-	journal->j_shrinker->batch = journal->j_max_transaction_buffers;
 	journal->j_shrinker->private_data = journal;
 
 	shrinker_register(journal->j_shrinker);
Signature · 1 line
2.35.3
Reviewed-by on v2 4/4
#16 lore raw

Re: [PATCH v2 4/4] jbd2: Drop pointless shrinker batch initialization

Quoted 7 lines · Jan Kara

In jbd2_journal_init_common() we set batch size of a shrinker shrinking checkpointed buffers to journal->j_max_transaction_buffers. But that is guaranteed to be 0 at that point so we effectively stay with the default shrinker batch size of 128. It has been like this since introduction of jbd2 shrinkers so just drop the pointless initialization.

Looks good to me.

Quoted 17 lines
---
 fs/jbd2/journal.c | 1 -
 1 file changed, 1 deletion(-)
fs/jbd2/journal.c +0 -1
index ae5b544ed0cc..c356cc027ed7 100644
@@ -1641,7 +1641,6 @@ static journal_t *journal_init_common(struct block_device *bdev,
 
 	journal->j_shrinker->scan_objects = jbd2_journal_shrink_scan;
 	journal->j_shrinker->count_objects = jbd2_journal_shrink_count;
-	journal->j_shrinker->batch = journal->j_max_transaction_buffers;
 	journal->j_shrinker->private_data = journal;
 
 	shrinker_register(journal->j_shrinker);