Skip to contentSkip to conversation

ocfs2: call journal flush to mark journal as empty after journal recovery when mount

PATCH In review Kai Li 8 participants 13 messages Dec 17, 2019 – Jan 1, 2020 OCFS2

lore
Share & tools
lore

Kai Li

author
#0 lore raw

PATCH v3 1/1 ocfs2: call journal flush to mark journal as empty after journal recovery when mount

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed
7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

 1. 8257792         JBD2_FLAG_SAME_UUID
 2. 619             JBD2_FLAG_SAME_UUID
 3. 24772864        JBD2_FLAG_SAME_UUID
 4. 8257802         JBD2_FLAG_SAME_UUID
 5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
 fs/ocfs2/journal.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +8 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
 	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);
+		jbd2_journal_unlock_updates(journal->j_journal);
+		if (status < 0)
+			mlog_errno(status);
+	}
+
 	status = ocfs2_journal_toggle_dirty(osb, 1, replayed);
 	if (status < 0) {
 		mlog_errno(status);
Signature · 1 line
2.24.0.windows.2
Reviewed-by on v3 1/1
#1 lore raw
Quoted 82 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed
7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

 1. 8257792         JBD2_FLAG_SAME_UUID
 2. 619             JBD2_FLAG_SAME_UUID
 3. 24772864        JBD2_FLAG_SAME_UUID
 4. 8257802         JBD2_FLAG_SAME_UUID
 5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13
Quoted 6 lines

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

Quoted 24 lines
---
 fs/ocfs2/journal.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +8 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
 	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);
+		jbd2_journal_unlock_updates(journal->j_journal);
+		if (status < 0)
+			mlog_errno(status);
+	}
+
 	status = ocfs2_journal_toggle_dirty(osb, 1, replayed);
 	if (status < 0) {
 		mlog_errno(status);
#2 lore raw
Quoted 8 lines · On Tue, 17 Dec 2019 10:12:14 +0800 Joseph Qi <[email protected]> wrote:
Quoted 6 lines

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

Do we think this fix should be backported into -stable kernels?

maintainer
#3 lore raw
Quoted 7 lines · Andrew Morton
Quoted 5 lines · On Tue, 17 Dec 2019 10:12:14 +0800 Joseph Qi <[email protected]> wrote:
Quoted 5 lines

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Do we think this fix should be backported into -stable kernels?

Since this is not an usual usecase, we don't have to. It is enough to let it go into upstream.

Thanks,
Joseph

Reviewed-by on v3 1/1
#4 lore raw
In reply to Kai Li
Quoted 112 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed
7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

  1. 8257792         JBD2_FLAG_SAME_UUID
  2. 619             JBD2_FLAG_SAME_UUID
  3. 24772864        JBD2_FLAG_SAME_UUID
  4. 8257802         JBD2_FLAG_SAME_UUID
  5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
  fs/ocfs2/journal.c | 8 ++++++++
  1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +8 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
  	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);
+		jbd2_journal_unlock_updates(journal->j_journal);
+		if (status < 0)
+			mlog_errno(status);
+	}
+
  	status = ocfs2_journal_toggle_dirty(osb, 1, replayed);
  	if (status < 0) {
  		mlog_errno(status);
#5 lore raw
In reply to Kai Li
Quoted 97 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed
7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

 1. 8257792         JBD2_FLAG_SAME_UUID
 2. 619             JBD2_FLAG_SAME_UUID
 3. 24772864        JBD2_FLAG_SAME_UUID
 4. 8257802         JBD2_FLAG_SAME_UUID
 5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
 fs/ocfs2/journal.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +0 -0
index 1afe57f425a0..68ba354cf361 100644

fs/ocfs2/journal.c int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed) · line 1071

Quoted 6 lines
fs/ocfs2/journal.c +3 -0
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
 	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start' won't be reset, and I wonder if the problem still remains.

Thanks,
Jun

Quoted 9 lines
fs/ocfs2/journal.c +5 -0
+		jbd2_journal_unlock_updates(journal->j_journal);
+		if (status < 0)
+			mlog_errno(status);
+	}
+
 	status = ocfs2_journal_toggle_dirty(osb, 1, replayed);
 	if (status < 0) {
 		mlog_errno(status);
maintainer
#6 lore raw
Quoted 101 lines · piaojun
Quoted 98 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed
7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

 1. 8257792         JBD2_FLAG_SAME_UUID
 2. 619             JBD2_FLAG_SAME_UUID
 3. 24772864        JBD2_FLAG_SAME_UUID
 4. 8257802         JBD2_FLAG_SAME_UUID
 5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
 fs/ocfs2/journal.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +0 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
Quoted 9 lines

fs/ocfs2/journal.c int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed) · line 1071

Quoted 5 lines
fs/ocfs2/journal.c +3 -0
 	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start' won't be reset, and I wonder if the problem still remains.

Yes, but we don't want this to fail the mount process, instead we just log an error and system administrator should know the result.

Thanks,
Joseph

#7 lore raw
Quoted 114 lines · Joseph Qi
Quoted 101 lines · piaojun
Quoted 98 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed
7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

 1. 8257792         JBD2_FLAG_SAME_UUID
 2. 619             JBD2_FLAG_SAME_UUID
 3. 24772864        JBD2_FLAG_SAME_UUID
 4. 8257802         JBD2_FLAG_SAME_UUID
 5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
 fs/ocfs2/journal.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +0 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
Quoted 9 lines

fs/ocfs2/journal.c int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed) · line 1071

Quoted 5 lines
fs/ocfs2/journal.c +3 -0
 	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start' won't be reset, and I wonder if the problem still remains.

Quoted 3 lines

Yes, but we don't want this to fail the mount process, instead we just log an error and system administrator should know the result.

Thanks for your reply and I have another question about this issue. IMO the second trans is not complete as jbd2 sb has not been updated, so we do not need to replay it when mount again.

Jun

author
#8 lore raw
Quoted 112 lines · piaojun
Quoted 110 lines · Joseph Qi
Quoted 108 lines · piaojun
Quoted 97 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed
7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

 1. 8257792         JBD2_FLAG_SAME_UUID
 2. 619             JBD2_FLAG_SAME_UUID
 3. 24772864        JBD2_FLAG_SAME_UUID
 4. 8257802         JBD2_FLAG_SAME_UUID
 5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
 fs/ocfs2/journal.c | 8 ++++++++
 1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +0 -0
index 1afe57f425a0..68ba354cf361 100644

fs/ocfs2/journal.c int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed) · line 1071

Quoted 6 lines
fs/ocfs2/journal.c +3 -0
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
 	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start' won't be reset, and I wonder if the problem still remains.

Quoted 10 lines
Quoted 3 lines

Yes, but we don't want this to fail the mount process, instead we just log an error and system administrator should know the result.

Thanks for your reply and I have another question about this issue. IMO the second trans is not complete as jbd2 sb has not been updated, so we do not need to replay it when mount again.

Jun

I don't think so. The problem is that jbd2 sb should be updated to mark a new start after mount rather than whether trans committed later is complete or not.

In fact , the trans is complete too as the commit log described.

Thanks

#9 lore raw
Quoted 13 lines · Likai
Quoted 12 lines · piaojun
Quoted 11 lines · Joseph Qi
Quoted 10 lines · piaojun
Quoted 9 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed

Hi,

I wonder that in this step, does the function jbd2_journal_commit_transaction() return?
I understand only jbd2_journal_commit_transaction() return it means the transaction
is committed completely, and the s_seq can be updated in jbd2_journal_commit_transaction()->jbd2_update_log_tail(), so I don't know how
this scenario happened.

If emergency restart happens in before jbd2_journal_commit_transaction() return,
I think this transaction shouldn't be a valid transaction.

Can you explain where the specific function is executed?

Thanks,
Yiwen.

Quoted 116 lines
Quoted 90 lines
Quoted 90 lines
Quoted 90 lines
Quoted 81 lines

7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after journal is replayed, and seq 15 is the first valid commit, but first seq is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

  1. 8257792         JBD2_FLAG_SAME_UUID
  2. 619             JBD2_FLAG_SAME_UUID
  3. 24772864        JBD2_FLAG_SAME_UUID
  4. 8257802         JBD2_FLAG_SAME_UUID
  5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent with the value recorded in block 1(seq is 14), journal recovery will be terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
  fs/ocfs2/journal.c | 8 ++++++++
  1 file changed, 8 insertions(+)
fs/ocfs2/journal.c +0 -0
index 1afe57f425a0..68ba354cf361 100644

fs/ocfs2/journal.c int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed) · line 1071

Quoted 6 lines
fs/ocfs2/journal.c +3 -0
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal *journal, int local, int replayed)
 
  	ocfs2_clear_journal_error(osb->sb, journal->j_journal, osb->slot_num);
 
+	if (replayed) {
+		jbd2_journal_lock_updates(journal->j_journal);
+		status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start' won't be reset, and I wonder if the problem still remains.

Quoted 10 lines
Quoted 3 lines

Yes, but we don't want this to fail the mount process, instead we just log an error and system administrator should know the result.

Thanks for your reply and I have another question about this issue. IMO the second trans is not complete as jbd2 sb has not been updated, so we do not need to replay it when mount again.

Jun

I don't think so. The problem is that jbd2 sb should be updated to mark a new start after mount rather than whether trans committed later is complete or not.

In fact , the trans is complete too as the commit log described.

Thanks

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

.

#10 lore raw
Quoted 35 lines · Jiangyiwen
Quoted 13 lines · Likai
Quoted 12 lines · piaojun
Quoted 11 lines · Joseph Qi
Quoted 10 lines · piaojun
Quoted 9 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed

Hi,

I wonder that in this step, does the function jbd2_journal_commit_transaction() return?
I understand only jbd2_journal_commit_transaction() return it means the transaction
is committed completely, and the s_seq can be updated in jbd2_journal_commit_transaction()->jbd2_update_log_tail(), so I don't know how
this scenario happened.

Quoted 1 line

If emergency restart happens in before jbd2_journal_commit_transaction()

The trick is that jbd2 kernel thread even doesn't have a chance to to update tail. Assuming that a timepoint sits between *mount completion* and the *first iteration* of committing work that jbd2 thread.

-Changwei

Quoted 137 lines

return,
I think this transaction shouldn't be a valid transaction.

Can you explain where the specific function is executed?

Thanks,
Yiwen.

Quoted 128 lines
Quoted 105 lines
Quoted 101 lines
Quoted 87 lines
Quoted 87 lines

7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it
is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch
does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after
journal is replayed, and seq 15 is the first valid commit, but first seq
is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0?? Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096?? Total Blocks: 32768?? First Block: 1 First Commit ID: 13?? Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1?? Dynamic Superblk Blknum: 0
Per Txn Block Limit??? Journal: 0??? Data: 0

Block 1: Journal Commit Block
Seq: 14?? Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15?? Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum??????? Flags
? 0. 587???????????? none
UUID: 00000000000000000000000000000000
? 1. 8257792???????? JBD2_FLAG_SAME_UUID
? 2. 619???????????? JBD2_FLAG_SAME_UUID
? 3. 24772864??????? JBD2_FLAG_SAME_UUID
? 4. 8257802???????? JBD2_FLAG_SAME_UUID
? 5. 513???????????? JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG
...
Block 7: Inode
Inode: 8257802?? Mode: 0640?? Generation: 57157641 (0x3682809) FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000?? ECC: 0000
Type: Regular?? Attr: 0x0?? Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root)?? Group: 0 (root)?? Size: 7
Links: 1?? Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec? 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec? 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec? 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan? 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15?? Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2 journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not 
unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): 

JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent
with the value recorded in block 1(seq is 14), journal recovery will be
terminated before seq 15 even though it is an unbroken commit, inode 8257802 is a new file and it will be lost.

---
? fs/ocfs2/journal.c | 8 ++++++++
? 1 file changed, 8 insertions(+)

Quoted 14 lines

fs/ocfs2/journal.c int ocfs2_journal_load(struct ocfs2_journal · line 1071

Quoted 11 lines
fs/ocfs2/journal.c +3 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct ocfs2_journal 
 *journal, int local, int replayed)
 ????? ocfs2_clear_journal_error(osb->sb, journal->j_journal, 
 osb->slot_num);
+??? if (replayed) {
+??????? jbd2_journal_lock_updates(journal->j_journal);
+??????? status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start' won't be reset, and I wonder if the problem still remains.

Quoted 4 lines

Yes, but we don't want this to fail the mount process, instead we just log
an error and system administrator should know the result.

Quoted 7 lines

Thanks for your reply and I have another question about this issue. IMO the second trans is not complete as jbd2 sb has not been updated, so we do not need to replay it when mount again.

Jun

I don't think so. The problem is that jbd2 sb should be updated to mark a new start after mount rather than whether trans committed later is complete or not.

In fact , the trans is complete too as the commit log described.

Thanks

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

.

#11 lore raw
Quoted 38 lines · Changwei Ge
Quoted 35 lines · Jiangyiwen
Quoted 13 lines · Likai
Quoted 12 lines · piaojun
Quoted 11 lines · Joseph Qi
Quoted 10 lines · piaojun
Quoted 9 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed

Hi,

I wonder that in this step, does the function jbd2_journal_commit_transaction() return?
I understand only jbd2_journal_commit_transaction() return it means the transaction
is committed completely, and the s_seq can be updated in jbd2_journal_commit_transaction()->jbd2_update_log_tail(), so I don't know how
this scenario happened.

Quoted 6 lines
Quoted 1 line

If emergency restart happens in before jbd2_journal_commit_transaction()

The trick is that jbd2 kernel thread even doesn't have a chance to to update tail. Assuming that a timepoint sits between *mount completion* and the *first iteration* of committing work that jbd2 thread.

Hi Changwei,

So I think if *first iteration* of committing work in jbd2 kernel thread is not
completed, and then the transaction should not be considered a valid transaction.

In addition, since we use ordered mode in jbd2, filesystem only ensure metadata consistency not data consistency. In above scenario, the filesystem is still normal without any damage.

Thanks,
Yiwen.

Quoted 147 lines

-Changwei

Quoted 142 lines

return,
I think this transaction shouldn't be a valid transaction.

Can you explain where the specific function is executed?

Thanks,
Yiwen.

Quoted 133 lines
Quoted 108 lines
Quoted 104 lines
Quoted 89 lines
Quoted 89 lines

7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it
is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch
does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after
journal is replayed, and seq 15 is the first valid commit, but first seq
is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0 Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096 Total Blocks: 32768 First Block: 1
First Commit ID: 13 Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1 Dynamic Superblk Blknum: 0
Per Txn Block Limit Journal: 0 Data: 0

Block 1: Journal Commit Block
Seq: 14 Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15 Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum Flags
0. 587 none
UUID: 00000000000000000000000000000000

  1. 8257792         JBD2_FLAG_SAME_UUID
  2. 619             JBD2_FLAG_SAME_UUID
  3. 24772864        JBD2_FLAG_SAME_UUID
  4. 8257802         JBD2_FLAG_SAME_UUID
  5. 513             JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG

...
Block 7: Inode
Inode: 8257802 Mode: 0640 Generation: 57157641 (0x3682809)
FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000 ECC: 0000
Type: Regular Attr: 0x0 Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root) Group: 0 (root) Size: 7
Links: 1 Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15 Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2
journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not 
unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): 

JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent
with the value recorded in block 1(seq is 14), journal recovery will be
terminated before seq 15 even though it is an unbroken commit, inode
8257802 is a new file and it will be lost.

---
  fs/ocfs2/journal.c | 8 ++++++++
  1 file changed, 8 insertions(+)
Quoted 15 lines

fs/ocfs2/journal.c int ocfs2_journal_load(struct · line 1071

Quoted 11 lines
fs/ocfs2/journal.c +3 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct 
 ocfs2_journal *journal, int local, int replayed)
      ocfs2_clear_journal_error(osb->sb, journal->j_journal, 
 osb->slot_num);
+    if (replayed) {
+ jbd2_journal_lock_updates(journal->j_journal);
+        status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start'
won't be reset, and I wonder if the problem still remains.

Quoted 4 lines

Yes, but we don't want this to fail the mount process, instead we just log
an error and system administrator should know the result.

Quoted 9 lines

Thanks for your reply and I have another question about this issue. IMO
the second trans is not complete as jbd2 sb has not been updated, so we
do not need to replay it when mount again.

Jun

I don't think so. The problem is that jbd2 sb should be updated to mark a new start after mount rather than whether trans committed later is complete or not.

In fact , the trans is complete too as the commit log described.

Thanks

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

.

.

#12 lore raw

PATCH v3 1/1 ocfs2: call journal flush to mark journal as empty after journal recovery when mount

Hi Yiwen,

Sorry for late reply since I was on a vocation last week :-)

Quoted 48 lines · Jiangyiwen
Quoted 38 lines · Changwei Ge
Quoted 35 lines · Jiangyiwen
Quoted 13 lines · Likai
Quoted 12 lines · piaojun
Quoted 11 lines · Joseph Qi
Quoted 10 lines · piaojun
Quoted 9 lines · Kai Li

If journal is dirty when mount, it will be replayed but jbd2 sb log tail cannot be updated to mark a new start because journal->j_flag has already been set with JBD2_ABORT first in journal_init_common. When a new transaction is committed, it will be recored in block 1 first(journal->j_tail is set to 1 in journal_reset).If emergency restart happens again before journal super block is updated unfortunately, the new recorded trans will not be replayed in the next mount.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

The following steps describe this procedure in detail.
1. mount and touch some files
2. these transactions are committed to journal area but not checkpointed
3. emergency restart
4. mount again and its journals are replayed
5. journal super block's first s_start is 1, but its s_seq is not updated
6. touch a new file and its trans is committed but not checkpointed

Hi,

I wonder that in this step, does the function jbd2_journal_commit_transaction() return?
I understand only jbd2_journal_commit_transaction() return it means the transaction
is committed completely, and the s_seq can be updated in jbd2_journal_commit_transaction()->jbd2_update_log_tail(), so I don't know how
this scenario happened.

Quoted 6 lines
Quoted 1 line

If emergency restart happens in before jbd2_journal_commit_transaction()

The trick is that jbd2 kernel thread even doesn't have a chance to to update tail. Assuming that a timepoint sits between *mount completion* and the *first iteration* of committing work that jbd2 thread.

Quoted 6 lines

Hi Changwei,

So I think if *first iteration* of committing work in jbd2 kernel thread is not
completed, and then the transaction should not be considered a valid transaction.

Um...
Here I think we are talking about *log tail* update. If jbd2 kernel thread already works for more than one iteration, it might have a chance to update log tail during committing trans. But if even no once of work iteration, it definitely can't update that tail to mark the boundary of uncheckpointed transactions.

Quoted 5 lines

In addition, since we use ordered mode in jbd2, filesystem only ensure metadata consistency not data consistency. In above scenario, the filesystem
is still normal without any damage.

True, the filesystem meta should be consistent. But the transaction is actually committed and valid. Relaying it does no harm.

In fact, I think it's a normal/standard use case of jbd2 that call jbd2_journal_flush() after relaying the entire journal. Please refer to ext4 and peer-help recovery progress in ocfs2.

Thanks,
Changwei

Quoted 152 lines

Thanks,
Yiwen.

Quoted 147 lines

??? -Changwei

Quoted 142 lines

return,
I think this transaction shouldn't be a valid transaction.

Can you explain where the specific function is executed?

Thanks,
Yiwen.

Quoted 133 lines
Quoted 108 lines
Quoted 104 lines
Quoted 89 lines
Quoted 89 lines

7. emergency restart again
8. mount and journal is dirty, but trans committed in 6 will not be replayed.

This exception happens easily when this lun is used by only one node. If it
is used by multi-nodes, other node will replay its journal and its journal super block will be updated after recovery like what this patch
does.

ocfs2_recover_node->ocfs2_replay_journal.

The following jbd2 journal can be generated by touching a new file after
journal is replayed, and seq 15 is the first valid commit, but first seq
is 13 in journal super block.
logdump:
Block 0: Journal Superblock
Seq: 0?? Type: 4 (JBD2_SUPERBLOCK_V2)
Blocksize: 4096?? Total Blocks: 32768?? First Block: 1 First Commit ID: 13?? Start Log Blknum: 1
Error: 0
Feature Compat: 0
Feature Incompat: 2 block64
Feature RO compat: 0
Journal UUID: 4ED3822C54294467A4F8E87D2BA4BC36
FS Share Cnt: 1?? Dynamic Superblk Blknum: 0
Per Txn Block Limit??? Journal: 0??? Data: 0

Block 1: Journal Commit Block
Seq: 14?? Type: 2 (JBD2_COMMIT_BLOCK)

Block 2: Journal Descriptor
Seq: 15?? Type: 1 (JBD2_DESCRIPTOR_BLOCK)
No. Blocknum??????? Flags
? 0. 587???????????? none
UUID: 00000000000000000000000000000000
? 1. 8257792???????? JBD2_FLAG_SAME_UUID
? 2. 619???????????? JBD2_FLAG_SAME_UUID
? 3. 24772864??????? JBD2_FLAG_SAME_UUID
? 4. 8257802???????? JBD2_FLAG_SAME_UUID
? 5. 513???????????? JBD2_FLAG_SAME_UUID JBD2_FLAG_LAST_TAG
...
Block 7: Inode
Inode: 8257802?? Mode: 0640?? Generation: 57157641 (0x3682809) FS Generation: 2839773110 (0xa9437fb6)
CRC32: 00000000?? ECC: 0000
Type: Regular?? Attr: 0x0?? Flags: Valid
Dynamic Features: (0x1) InlineData
User: 0 (root)?? Group: 0 (root)?? Size: 7
Links: 1?? Clusters: 0
ctime: 0x5de5d870 0x11104c61 -- Tue Dec? 3 11:37:20.286280801 2019
atime: 0x5de5d870 0x113181a1 -- Tue Dec? 3 11:37:20.288457121 2019
mtime: 0x5de5d870 0x11104c61 -- Tue Dec? 3 11:37:20.286280801 2019
dtime: 0x0 -- Thu Jan? 1 08:00:00 1970
...
Block 9: Journal Commit Block
Seq: 15?? Type: 2 (JBD2_COMMIT_BLOCK)

The following is jouranl recovery log when recovering the upper jbd2
journal when mount again.
syslog:

[ 2265.648622] ocfs2: File system on device (252,1) was not 
unmounted cleanly, recovering it.
[ 2265.649695] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 0
[ 2265.650407] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 1
[ 2265.650409] fs/jbd2/recovery.c:(do_one_pass, 449): Starting 
recovery pass 2
[ 2265.650410] fs/jbd2/recovery.c:(jbd2_journal_recover, 278): 

JBD2: recovery, exit status 0, recovered transactions 13 to 13

Due to first commit seq 13 recorded in journal super is not consistent
with the value recorded in block 1(seq is 14), journal recovery will be
terminated before seq 15 even though it is an unbroken commit, inode
8257802 is a new file and it will be lost.

---
? fs/ocfs2/journal.c | 8 ++++++++
? 1 file changed, 8 insertions(+)

Quoted 15 lines

fs/ocfs2/journal.c int ocfs2_journal_load(struct · line 1071

Quoted 11 lines
fs/ocfs2/journal.c +3 -0
index 1afe57f425a0..68ba354cf361 100644
@@ -1066,6 +1066,14 @@ int ocfs2_journal_load(struct 
 ocfs2_journal *journal, int local, int replayed)
 ????? ocfs2_clear_journal_error(osb->sb, journal->j_journal, 
 osb->slot_num);
+??? if (replayed) {
+ jbd2_journal_lock_updates(journal->j_journal);
+??????? status = jbd2_journal_flush(journal->j_journal);

What if jbd2_journal_flush gets failed? The 's_sequence' and 's_start'
won't be reset, and I wonder if the problem still remains.

Quoted 4 lines

Yes, but we don't want this to fail the mount process, instead we just log
an error and system administrator should know the result.

Quoted 9 lines

Thanks for your reply and I have another question about this issue. IMO
the second trans is not complete as jbd2 sb has not been updated, so we
do not need to replay it when mount again.

Jun

I don't think so. The problem is that jbd2 sb should be updated to mark a new start after mount rather than whether trans committed later is complete or not.

In fact , the trans is complete too as the commit log described.

Thanks

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

.

.