Skip to contentSkip to conversation

ext4 fsck vs. kernel recovery policy

RFC New dann frazier 5 participants 11 messages Aug 27, 2019 – Jan 27, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

hey,
I'm curious if there's a policy about what types of unclean
shutdowns 'e2fsck -p' can recover, vs. what the kernel will automatically recover on mount. We're seeing that unclean shutdowns w/ data=journal,journal_csum frequently result in invalid checksums that causes the kernel to abort recovery, while 'e2fsck -p' resolves the issue non-interactively.

Driver for this question is that some Ubuntu installs set fstab's passno=0 for the root fs - which I'm told is based on the assumption that both kernel & e2fsck -p have parity when it comes to automatic recovery - that's obviously does not appear to be the case - but I wanted to confirm whether or not that is by design.

-dann

#1 lore raw
Quoted 8 lines · dann frazier

hey,
I'm curious if there's a policy about what types of unclean
shutdowns 'e2fsck -p' can recover, vs. what the kernel will automatically recover on mount. We're seeing that unclean shutdowns w/ data=journal,journal_csum frequently result in invalid checksums that causes the kernel to abort recovery, while 'e2fsck -p' resolves the issue non-interactively.

The kernel journal recovery will only replay the journal blocks. It doesn't do any check and repair of filesystem correctness. During and after e2fsck replays the journal blocks it still does basic correctness checking, and if an error is found it will fall back to a full scan.

Quoted 5 lines

Driver for this question is that some Ubuntu installs set fstab's passno=0 for the root fs - which I'm told is based on the assumption that both kernel & e2fsck -p have parity when it comes to automatic recovery - that's obviously does not appear to be the case - but I wanted to confirm whether or not that is by design.

The first thing to figure out is why there are errors with the journal blocks. That can cause problems for both the kernel and e2fsck journal replay.

Using data=journal is not a common option, so it is likely that the issue relates to this. IMHO, using data=journal could be helpful for small file writes and/or sync IO, but there have been discussions lately about removing this functionality. If you have some use case that shows real improvements with data=journal, please let us know.

Cheers, Andreas

#2 lore raw
Quoted 10 lines · Andreas Dilger
Quoted 8 lines · dann frazier

hey,
I'm curious if there's a policy about what types of unclean
shutdowns 'e2fsck -p' can recover, vs. what the kernel will automatically recover on mount. We're seeing that unclean shutdowns w/ data=journal,journal_csum frequently result in invalid checksums that causes the kernel to abort recovery, while 'e2fsck -p' resolves the issue non-interactively.

Quoted 4 lines

The kernel journal recovery will only replay the journal blocks. It doesn't do any check and repair of filesystem correctness. During and after e2fsck replays the journal blocks it still does basic correctness checking, and if an error is found it will fall back to a full scan.

hey Andreas!

Here's a log to clarify what I'm seeing:

$ sudo mount /dev/nbd0 mnt
JBD2: Invalid checksum recovering data block 517634 in log
JBD2: Invalid checksum recovering data block 517633 in log
[...]
JBD2: Invalid checksum recovering data block 517004 in log
JBD2: Invalid checksum recovering data block 4915712 in log
JBD2: recovery failed
EXT4-fs (nbd0): error loading journal
mount: /tmp/mnt: can't read superblock on /dev/nbd0.
$ sudo e2fsck -p /dev/nbd0 
/dev/nbd0: recovering journal
JBD2: Invalid checksum recovering block 517732 in log
JBD2: Invalid checksum recovering block 517519 in log
[...]
JBD2: Invalid checksum recovering block 4915712 in log
Journal checksum error found in /dev/nbd0
/dev/nbd0: Clearing orphaned inode 128798 (uid=0, gid=0, mode=040600, size=4096)
/dev/nbd0: Clearing orphaned inode 514998 (uid=0, gid=0, mode=040600, size=4096)
[...]
/dev/nbd0: Clearing orphaned inode 774759 (uid=0, gid=0, mode=0100600, size=4096)
/dev/nbd0 was not cleanly unmounted, check forced.
/dev/nbd0: 2127984/2195456 files (0.0% non-contiguous), 2963178/8780544 blocks

So is it correct to say that the checksum errors were identifying filesystem correctness issues, and therefore e2fsck was needed to correct them?

Quoted 6 lines
Quoted 5 lines

Driver for this question is that some Ubuntu installs set fstab's passno=0 for the root fs - which I'm told is based on the assumption that both kernel & e2fsck -p have parity when it comes to automatic recovery - that's obviously does not appear to be the case - but I wanted to confirm whether or not that is by design.

Quoted 6 lines

The first thing to figure out is why there are errors with the journal blocks. That can cause problems for both the kernel and e2fsck journal replay.

Using data=journal is not a common option, so it is likely that the issue relates to this.

You're probably right - this issue is very easy to reproduce w/ data=journal,journal_checksum. I was never able to reproduce it otherwise.

Quoted 4 lines

IMHO, using data=journal could be helpful for
small file writes and/or sync IO, but there have been discussions lately about removing this functionality. If you have some use case that shows real improvements with data=journal, please let us know.

I don't have such a use case myself. The issue was reported by a user, and it got me wondering about the basis for our passno=0 default.

-dann

maintainer
#3 lore raw

(Changing the cc from linux-fsdevel to linux-ext4.)

Quoted 5 lines · dann frazier

JBD2: Invalid checksum recovering data block 517634 in log

So is it correct to say that the checksum errors were identifying filesystem correctness issues, and therefore e2fsck was needed to correct them?

That's correct. More precisely, checksum errors for journal blocks are presumed to mean that file system might be corrupt, so a full e2fsck check was needed to make sure the file system was consistent.

Quoted 3 lines

You're probably right - this issue is very easy to reproduce w/ data=journal,journal_checksum. I was never able to reproduce it otherwise.

I've looked at the data block numbers that you've reported, and they come from a journald file. The problem is with data=journal + journal_checksum + mmap. Unfortunately, we don't handle that combination correctly at the moment.

The fix is going to have to involve fixing __ext4_journalled_writepage() to call set_page_writeback() before it unlocks the page, adding a list of pages under data=journalled writeback which is attached to the transaction handle, have the jbd2 commit hook call end_page_writeback() on all of these pages, and then in the places where ext4 calls wait_for_stable_page() or grab_cache_page_write_begin(),
we need to add:

	if (ext4_should_journal_data(inode))
		wait_on_page_writeback(page);

It's all relatively straightforward except for the part where we have to attach a list of pages to the currently running transaction. That will require adding some plumbing into the jbd2 layer.

Dann, any interest in trying to code this fix?

- Ted

#4 lore raw
Quoted 30 lines · Theodore Y. Ts'o

(Changing the cc from linux-fsdevel to linux-ext4.)

Quoted 5 lines · dann frazier

JBD2: Invalid checksum recovering data block 517634 in log

So is it correct to say that the checksum errors were identifying filesystem correctness issues, and therefore e2fsck was needed to correct them?

That's correct. More precisely, checksum errors for journal blocks are presumed to mean that file system might be corrupt, so a full e2fsck check was needed to make sure the file system was consistent.

Quoted 3 lines

You're probably right - this issue is very easy to reproduce w/ data=journal,journal_checksum. I was never able to reproduce it otherwise.

I've looked at the data block numbers that you've reported, and they come from a journald file. The problem is with data=journal + journal_checksum + mmap. Unfortunately, we don't handle that combination correctly at the moment.

The fix is going to have to involve fixing __ext4_journalled_writepage() to call set_page_writeback() before it unlocks the page, adding a list of pages under data=journalled writeback which is attached to the transaction handle, have the jbd2 commit hook call end_page_writeback() on all of these pages, and then in the places where ext4 calls wait_for_stable_page() or grab_cache_page_write_begin(),
we need to add:

Quoted 8 lines
	if (ext4_should_journal_data(inode))
		wait_on_page_writeback(page);

It's all relatively straightforward except for the part where we have to attach a list of pages to the currently running transaction. That will require adding some plumbing into the jbd2 layer.

Dann, any interest in trying to code this fix?

Thanks Ted. I've the interest, I'll see if I can find the time :)

-dann

#5 lore raw
In reply to Theodore Y. Ts'o

From: Mauricio Faria de Oliveira <[email protected]>

Quoted 1 line · Theodore Y. Ts'o

Dann, any interest in trying to code this fix?

Hi Ted,

Dann and I discussed this a little bit, and I could try to code this fix.

It's working for the most part (i.e., no regressions on xfstests, and it runs our test-case / stress-ng mmap/file test without issues), but I had a few questions about a couple of deadlocks I've hit/fixed (that I think may still happen on other paths), and your opinion/advice on the code.

I'm sending the RFC patch too (on v5.5-rc2) to describe these questions. It's clearly not final. Any comments/feedback would be much appreciated!

Apologies for the long email upfront; hope the extra context does help.

Quoted 2 lines · Theodore Y. Ts'o

The fix is going to have to involve fixing __ext4_journalled_writepage() to call set_page_writeback() before it unlocks the page, <snip>

Q1: That's the page_unlock() in 'out:', right?

Otherwise (in page_unlock() after get_page()) the code hits a deadlock in the next page_lock(), against write_cache_pages(), because it calls page_lock() and then wait_on_page_writeback().

Q2: Must set_page_writeback() occur before ext4_journal_stop() ?

I think so, to prevent a race condition between the journal commit hook running shortly after ext4_journal_stop() trying to end_page_writeback() before set_page_writeback() runs (which hits a BUG()),
but wondered if any guarantees exist that could relax this case/order.

Quoted 4 lines

<snip> adding a list of
pages under data=journalled writeback which is attached to the transaction handle, have the jbd2 commit hook call end_page_writeback() on all of these pages, <snip>

Q3: Instead of coding something new for this, I re-used something, but it seems to be relatively more overhead -- what do you think?

I relied on 'struct ext4_journal_cb_entry' and ext4_journal_callback_add(), which adds such callback entries in handle->h_transaction->t_private_list.

Those entries are called back from ext4_journal_commit_callback(), which is 'sbi->s_journal->j_commit_callback' called by jbd2_journal_commit_transaction().

This is relatively more costly/higher overhead compared with your suggestion:

i.e., allocates memory for callback entry, and on the commit hook loops over a list of callback entries and call their function pointers on their private data field / page pointer, then call end_page_writeback() and kfree() entry;

vs. still allocating memory for a callback entry item (just a page pointer) but then just calling one function to go over a list of page pointers, and call end_page_writeback() on them.

Quoted 7 lines

<snip> and then in the places where ext4 calls wait_for_stable_page() or grab_cache_page_write_begin(), we need to add:

	if (ext4_should_journal_data(inode))
		wait_on_page_writeback(page);

Q4: When done between ext4_journal_start() and ext4_journal_stop() this hit a deadlock with jbd2_journal_commit_transaction(), described below.

I wonder if this is due to an implementation/logic error on my code. If you could please provide some detail about it, that'd be helpful.

The deadlock is:

Task 1) stress-ng: calls jbd2_journal_force_commit().

Task 2) jbd2 task: calls jbd2_journal_commit_transaction(),
which is waiting on 'write_lock(&j_state_lock)' in the
'while (atomic_read(&commit_transaction->t_updates))' loop.

If I understand it correctly, this is waiting on updates to the transaction handle to finish, ie they called ext4_journal_start() but _not yet_ ext4_journal_stop() (where jbd2 drops 't_updates').

Then there is,

Task 3) stress-ng: calls ext4_write_begin() and is stopped at
wait_on_page_writeback() between ext4_journal_start()/_stop(), right after wait_for_stable_page(), per the algorithm described.

(all stack traces provided later in the email)

So wait_on_page_writeback() waits for jbd2_journal_commit_transaction() to eventually callback end_page_writeback() - but that is waiting for wait_on_page_writeback() to finish and then call ext4_journal_stop().

That deadlock could be fixed by moving the wait_on_page_writeback() before ext4_journal_start().

But since wait_for_stable_page() is already in place, and it can call wait_on_page_writeback() too and thus hit the same deadlock,

I wonder if this issue is more general and has to be fixed for this implementation, or this implementation has to change to adapt to the existing logic, and avoid this issue.

Quoted 3 lines

It's all relatively straightforward except for the part where we have to attach a list of pages to the currently running transaction. That will require adding some plumbing into the jbd2 layer.

There was this other issue I had to work around, again not sure if it's a problem on my code, or something that actually needs to be considered.

If set_page_writeback() is used on the msync() / ext4_sync_file() path, there's a delay of seconds in file_write_and_wait_range() as it calls wait_on_page_writeback(), until the periodic journal commit clears it.

msync()
-> ext4_sync_file()
-> file_write_and_wait_range()

      -> __filemap_fdatawrite_range()
         -> ...
            -> __ext4_journalled_writepage()
      -> __filemap_fdatawait_range()
         -> wait_on_page_writeback()

So, I added a 'sync' parameter to __ext4_journalled_writepage(), and it's only true in that path (checks for (wbc->sync_mode == WB_SYNC_ALL)).

Q5: Is it okay to skip set/end_page_writeback() for this path, since ext4_sync_file() already calls ext4_force_commit() for data=journal?

Q6: Or should __ext4_journalled_writepage() use ext4_handle_sync() in the sync case, or maybe on both sync/async cases anyway?

Thank you,
Mauricio

Test-case:
---

# mkfs.ext4 -F $DEV && \
  mount -o data=journal,journal_checksum $DEV $PATH && \
  cd $PATH && \
  stress-ng --mmap 512 --mmap-file -t 1w

Stack traces from Q4:
---

Task 1)
comm stress-ng-mmap state D

[<0>] jbd2_log_wait_commit+0xa7/0x110
[<0>] __jbd2_journal_force_commit+0x53/0x90
[<0>] jbd2_journal_force_commit+0x18/0x30
[<0>] ext4_sync_file+0x140/0x3c0
[<0>] __x64_sys_msync+0x193/0x200
[<0>] do_syscall_64+0x43/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Task 2)
comm jbd2/vdc-8 state D

[<0>] jbd2_journal_commit_transaction+0x242/0x187f
[<0>] kjournald2+0xb2/0x280
[<0>] kthread+0xf6/0x130
[<0>] ret_from_fork+0x35/0x40

jbd2_journal_commit_transaction+0x242 is:

while (atomic_read(&commit_transaction->t_updates)) {
...

	if (atomic_read(&commit_transaction->t_updates)) {
	...
	write_lock(&journal->j_state_lock); // <<-- HERE.

Task 3)
comm stress-ng-mmap state D

[<0>] wait_on_page_bit+0xfe/0x1c0
[<0>] ext4_write_begin+0x215/0x640
[<0>] generic_perform_write+0xba/0x1c0
[<0>] __generic_file_write_iter+0x143/0x1c0
[<0>] ext4_file_write_iter+0xdc/0x380
[<0>] new_sync_write+0x116/0x1b0
[<0>] vfs_write+0xb1/0x190
[<0>] ksys_write+0x5a/0xd0
[<0>] do_syscall_64+0x43/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

ext4_write_begin+0x215/0x640 is:

retry_journal:
...

        handle = ext4_journal_start(...)
...
        /* In case writeback began while the page was unlocked */
        wait_for_stable_page(page);
        if (ext4_should_journal_data(inode))
                wait_on_page_writeback(page); // <<-- HERE.

...

Other tasks:

comm stress-ng-mmap state D

[<0>] wait_transaction_locked+0x82/0xc0
[<0>] add_transaction_credits+0xd5/0x290
[<0>] start_this_handle+0xf6/0x450
[<0>] jbd2__journal_start+0xd7/0x1e0

...
(from: syscall, pagefault)

Mauricio Faria de Oliveira (1):
ext4: set page writeback on journalled writepage

 fs/ext4/ext4_jbd2.h | 11 ++++++
 fs/ext4/inline.c    | 13 +++++++
 fs/ext4/inode.c     | 82 ++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 102 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
#6 lore raw

RFC 1/1 ext4: set page writeback on journalled writepage

From: Mauricio Faria de Oliveira <[email protected]>

Work in progress / request for comments to fix issue with journal consistency errors on unclean shutdown with mmap() on ext4 data=journal,journal_checksum mode.

Reference:
https://lore.kernel.org/linux-ext4/[email protected]/

---
 fs/ext4/ext4_jbd2.h | 11 ++++++
 fs/ext4/inline.c    | 13 +++++++
 fs/ext4/inode.c     | 82 ++++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 102 insertions(+), 4 deletions(-)
fs/ext4/ext4_jbd2.h +11 -0
index a6b9b66dbfad..8b51ca8231b7 100644
@@ -144,6 +144,17 @@ struct ext4_journal_cb_entry {
 	/* user data goes here */
 };
 
+/**
+ * struct ext4_journal_cb_entry_simple - Simple structure for callback information.
+ *
+ * This struct is a 'simple' structure on top of the base/seed structure,
+ * which adds a private data pointer to be used by the callback function.
+ */
+struct ext4_journal_cb_entry_simple {
+	struct ext4_journal_cb_entry jce;
+	void *jce_private;
+};
+
 /**
  * ext4_journal_callback_add: add a function to call after transaction commit
  * @handle: active journal transaction handle to register callback on
fs/ext4/inline.c +13 -0
index 2fec62d764fa..a168fe497d5d 100644
@@ -565,6 +565,9 @@ static int ext4_convert_inline_data_to_extent(struct address_space *mapping,
 		ret = -ENOMEM;
 		goto out;
 	}
+	/* XXX(mfo): deadlock with journal? */
+	if (ext4_should_journal_data(inode))
+		wait_on_page_writeback(page);
 
 	ext4_write_lock_xattr(inode, &no_expand);
 	sem_held = 1;
@@ -693,6 +696,9 @@ int ext4_try_to_write_inline_data(struct address_space *mapping,
 		ret = -ENOMEM;
 		goto out;
 	}
+	/* XXX(mfo): deadlock with journal? */
+	if (ext4_should_journal_data(inode))
+		wait_on_page_writeback(page);
 
 	*pagep = page;
 	down_read(&EXT4_I(inode)->xattr_sem);
@@ -808,6 +814,10 @@ static int ext4_da_convert_inline_data_to_extent(struct address_space *mapping,
 	page = grab_cache_page_write_begin(mapping, 0, flags);
 	if (!page)
 		return -ENOMEM;
+	/* XXX(mfo): should not deadlock with journal;
+	 * this is only called after ext4_journal_stop(). */
+	if (ext4_should_journal_data(inode))
+		wait_on_page_writeback(page);
 
 	down_read(&EXT4_I(inode)->xattr_sem);
 	if (!ext4_has_inline_data(inode)) {
@@ -911,6 +921,9 @@ int ext4_da_write_inline_data_begin(struct address_space *mapping,
 		ret = -ENOMEM;
 		goto out_journal;
 	}
+	/* XXX(mfo): deadlock with journal? */
+	if (ext4_should_journal_data(inode))
+		wait_on_page_writeback(page);
 
 	down_read(&EXT4_I(inode)->xattr_sem);
 	if (!ext4_has_inline_data(inode)) {
fs/ext4/inode.c +78 -4

156 lines not on this page · raw message

Signature · 1 line
2.17.1
#7 lore raw
In reply to Theodore Y. Ts'o
Quoted 28 lines · Theodore Y. Ts'o
Quoted 5 lines · dann frazier

JBD2: Invalid checksum recovering data block 517634 in log

So is it correct to say that the checksum errors were identifying filesystem correctness issues, and therefore e2fsck was needed to correct them?

That's correct. More precisely, checksum errors for journal blocks are presumed to mean that file system might be corrupt, so a full e2fsck check was needed to make sure the file system was consistent.

Quoted 3 lines

You're probably right - this issue is very easy to reproduce w/ data=journal,journal_checksum. I was never able to reproduce it otherwise.

I've looked at the data block numbers that you've reported, and they come from a journald file. The problem is with data=journal + journal_checksum + mmap. Unfortunately, we don't handle that combination correctly at the moment.

The fix is going to have to involve fixing __ext4_journalled_writepage() to call set_page_writeback() before it unlocks the page, adding a list of pages under data=journalled writeback which is attached to the transaction handle, have the jbd2 commit hook call end_page_writeback() on all of these pages, and then in the places where ext4 calls wait_for_stable_page() or grab_cache_page_write_begin(),
we need to add:

Quoted 3 lines
	if (ext4_should_journal_data(inode))
		wait_on_page_writeback(page);

If I understood and wrote things correctly, this can hit a deadlock between

1) jbd2_journal_commit_transaction() -- waiting for t_updates to drop (i.e., ext4_journal_stop() to be called), blocking commit / end_page_writeback().

and

2) ext4_write_begin() -- waiting in wait_on_page_writeback() in between the calls to ext4_journal_start()/stop(), blocking t_updates from dropping.

I worked around it moving wait_on_page_writeback() before ext4_journal_start(), but wonder if this is really expected and the algorithm needs tuning/changes, or not expected and thus an implementation error. (First time with ext4 code.)

P.S.: sorry to bother again about it (I already wrote/pinged on [1, 2]) but this does seem interesting and shows an issue we're interested in fixing up.

Thanks again for your help/suggestions on this.

[1] https://lore.kernel.org/linux-ext4/[email protected]/
[2] https://lore.kernel.org/linux-ext4/[email protected]/

cheers,
Mauricio

Quoted 7 lines

It's all relatively straightforward except for the part where we have to attach a list of pages to the currently running transaction. That will require adding some plumbing into the jbd2 layer.

Dann, any interest in trying to code this fix?

- Ted

#8 lore raw
In reply to dann frazier
Quoted 8 lines · dann frazier

hey,
I'm curious if there's a policy about what types of unclean
shutdowns 'e2fsck -p' can recover, vs. what the kernel will automatically recover on mount. We're seeing that unclean shutdowns w/ data=journal,journal_csum frequently result in invalid checksums that causes the kernel to abort recovery, while 'e2fsck -p' resolves the issue non-interactively.

Quoted 7 lines

Driver for this question is that some Ubuntu installs set fstab's passno=0 for the root fs - which I'm told is based on the assumption that both kernel & e2fsck -p have parity when it comes to automatic recovery - that's obviously does not appear to be the case - but I wanted to confirm whether or not that is by design.

-dann

Ted or others more involved w/ ext4 will speak w/ authority but it's my understanding that log replay, whether done by userspace or by the kernel, should always return the filesystem to a consistent state. If that's not the case, scripting things so that you grab a qcow-format e2image prior to fsck so that you can share the problematic image with developers may help.

(In XFS land, a large portion of the unreplayable logs we see are the result of storage that didn't /actually/ persist IOs that it claimed were persisted prior to the crash/poweroff.)

-Eric

#9 lore raw
Quoted 15 lines · Eric Sandeen
Quoted 14 lines · dann frazier

hey,
I'm curious if there's a policy about what types of unclean
shutdowns 'e2fsck -p' can recover, vs. what the kernel will automatically recover on mount. We're seeing that unclean shutdowns w/ data=journal,journal_csum frequently result in invalid checksums that causes the kernel to abort recovery, while 'e2fsck -p' resolves the issue non-interactively.

Driver for this question is that some Ubuntu installs set fstab's passno=0 for the root fs - which I'm told is based on the assumption that both kernel & e2fsck -p have parity when it comes to automatic recovery - that's obviously does not appear to be the case - but I wanted to confirm whether or not that is by design.

Quoted 5 lines
Quoted 1 line

-dann

Ted or others more involved w/ ext4 will speak w/ authority but it's my understanding that log replay, whether done by userspace or by the kernel, should always return the filesystem to a consistent state.

I should amend: "from an otherwise normal unclean shutdown"

corruption discovered during recovery is a different matter, as adilger pointed out.

-Eric

#10 lore raw
In reply to Eric Sandeen
Quoted 15 lines · Eric Sandeen
Quoted 14 lines · dann frazier

hey,
I'm curious if there's a policy about what types of unclean
shutdowns 'e2fsck -p' can recover, vs. what the kernel will automatically recover on mount. We're seeing that unclean shutdowns w/ data=journal,journal_csum frequently result in invalid checksums that causes the kernel to abort recovery, while 'e2fsck -p' resolves the issue non-interactively.

Driver for this question is that some Ubuntu installs set fstab's passno=0 for the root fs - which I'm told is based on the assumption that both kernel & e2fsck -p have parity when it comes to automatic recovery - that's obviously does not appear to be the case - but I wanted to confirm whether or not that is by design.

Quoted 8 lines
Quoted 1 line

-dann

Ted or others more involved w/ ext4 will speak w/ authority but it's my understanding that log replay, whether done by userspace or by the kernel, should always return the filesystem to a consistent state. If that's not the case, scripting things so that you grab a qcow-format e2image prior to fsck so that you can share the problematic image with developers may help.

Thanks Eric. I captured an image in case it's useful:

https://people.canonical.com/~dannf/md2.e2ic.qcow2

-dann

Quoted 6 lines

(In XFS land, a large portion of the unreplayable logs we see are the result of storage that didn't /actually/ persist IOs that it claimed were persisted prior to the crash/poweroff.)

-Eric