Error handling: How to "lose" a transaction

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Chris -

I'm starting to dig into the fun part of error handling and btrfs_commit_transaction is a minefield right now.

I've been thinking about how I would go about recovering from a serious error like an -EIO while writing out or an -ENOMEM in a deep part of the code that it's prohibitively expensive to recover from. Mostly I'm looking for the best way to make calling btrfs_std_error() be functionally equivalent to killing the power on the disk. We already block off new writers, but that's obviously nowhere near enough. We could have an open transaction floating around, uncommitted transactions queued, and then an unrecoverable error hits, forcing us to shut it all down.

It seems to me that that a similar method of recovery that I wrote for reiserfs can be used here as well. Am I understanding correctly that if I go through the motions of committing the transaction *except* for updating the tree roots, or maybe even doing that but declining to write the superblocks out, that the transaction essentially doesn't exist on disk? Including the allocations? The in-memory representation will not match what's on disk, but that's what happens with every file system in RO-failure mode. With CoW even for data, data is essentially frozen in time as well. (I suppose with nodatacow that's not true, but that's for another day.)

Quoted 18 lines · Jeff Mahoney

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hi Chris -

I'm starting to dig into the fun part of error handling and btrfs_commit_transaction is a minefield right now.

I've been thinking about how I would go about recovering from a serious error like an -EIO while writing out or an -ENOMEM in a deep part of the code that it's prohibitively expensive to recover from. Mostly I'm looking for the best way to make calling btrfs_std_error() be functionally equivalent to killing the power on the disk. We already block off new writers, but that's obviously nowhere near enough. We could have an open transaction floating around, uncommitted transactions queued, and then an unrecoverable error hits, forcing us to shut it all down.

Quoted 11 lines

It seems to me that that a similar method of recovery that I wrote for reiserfs can be used here as well. Am I understanding correctly that if I go through the motions of committing the transaction *except* for updating the tree roots, or maybe even doing that but declining to write the superblocks out, that the transaction essentially doesn't exist on disk? Including the allocations? The in-memory representation will not match what's on disk, but that's what happens with every file system in RO-failure mode. With CoW even for data, data is essentially frozen in time as well. (I suppose with nodatacow that's not true, but that's for another day.)

Hi Jeff,

Thanks for taking another pass at this.

It should be possible to just skip the step where we update the roots in the super and you'll keep a fully consistent FS on disk. The only rule would be that you're not allowed to take a block that we've freed in the aborted transaction and reuse it.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 30 lines · Chris Mason

Quoted 18 lines · Jeff Mahoney

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hi Chris -

I'm starting to dig into the fun part of error handling and btrfs_commit_transaction is a minefield right now.

I've been thinking about how I would go about recovering from a serious error like an -EIO while writing out or an -ENOMEM in a deep part of the code that it's prohibitively expensive to recover from. Mostly I'm looking for the best way to make calling btrfs_std_error() be functionally equivalent to killing the power on the disk. We already block off new writers, but that's obviously nowhere near enough. We could have an open transaction floating around, uncommitted transactions queued, and then an unrecoverable error hits, forcing us to shut it all down.

Quoted 11 lines

It seems to me that that a similar method of recovery that I wrote for reiserfs can be used here as well. Am I understanding correctly that if I go through the motions of committing the transaction *except* for updating the tree roots, or maybe even doing that but declining to write the superblocks out, that the transaction essentially doesn't exist on disk? Including the allocations? The in-memory representation will not match what's on disk, but that's what happens with every file system in RO-failure mode. With CoW even for data, data is essentially frozen in time as well. (I suppose with nodatacow that's not true, but that's for another day.)

Quoted 8 lines

Hi Jeff,

Thanks for taking another pass at this.

It should be possible to just skip the step where we update the roots in the super and you'll keep a fully consistent FS on disk. The only rule would be that you're not allowed to take a block that we've freed in the aborted transaction and reuse it.

Perfect.

Sorry I haven't responded to this yet. I started digging right in and I've started to have some good results. It turns out there's already a btrfs_cleanup_transaction call that will tear down outstanding transactions. It's not perfect and I've fixed a few bugs in there, but it saved me a bunch of effort. I just wished I noticed it a day before since I had it half implemented myself. :)

This afternoon I started running xfstests on a dm-linear mapped partition. Halfway through a sufficiently long test, I swap out the linear mapping to an error mapping. It still crashes, but somewhat less spectacularly. There are still a ton of BUG_ON's I need to eliminate as well as work out the usual I/O error-recovery issue of uninterruptible, unrecoverable writeback contexts and still-locked pages holding up exit. I'm pretty pleased with the results so far and am pretty optimistic.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJO8p0MAAoJEB57S2MheeWyF4YP/A6uUzuP4zui+iwenSP44trw FkONuTidZRjSgA4tXNrdsnIF2txtiewzp0HvWWudw5rnMNQzznyO0WynKHSPG3ep xFZnfpvaYoCaMQt70IxAQFDsZpowbPAI8194mbJqKAql4f2RNzlg/3fR4k+Fz6Ye Gu824uEbtyHghy96C37e/E30Zizu6+S7xrx8jwmnKbq44docoIV3Pw9LZGOU99Db 1IFipExd0Z/ZhTTiK4gZ787nPhM9QNfxw/9+h1g4gUfJqlcmRrcwGJmOj5iOBGBt Man51ZCI8hYBpubTgvTQalut+uLq9lCoBZQGTbKHLNLd21qM+Ji4KCAQzMBUtqGn pzSfs3Gdwa1WjYszINAS6gqA+0ubh1F/WxGwJKW85JnAYy8OjTJHru7GlYzt3C9Y gouU7xgrneVn+lZFwV9X0gwX8yLQx5Lh9YEF6AJLXJuXHg4zGZyhpFjVkmTlle93 dFUblB92q9lxdw5V8f1Uw+EDIlACZZRo7MFDSypjdTTryRFiAjhCtBdBpnu54Mrb fH2kdhPCBm4YqAQLlo43aOPAbkOYElAr0rgPvqaLzimZLAW0kd/nGU/if3mhMMa2 7ad7tKTQyktyGKuEkMPnSCU8SqFNGA750aeFG22uJJjbdCytyzkJmeqYQD5oykqm vDpKh0g20Fcqb98q+qbt
=jjDk
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 12/22/2011 10:59 AM, Jeff Mahoney wrote: Sorry I haven't responded to this yet. I started digging right in and I've started to have some good results. It turns out there's already a btrfs_cleanup_transaction call that will tear down outstanding transactions. It's not perfect and I've fixed a few bugs in there, but it saved me a bunch of effort. I just wished I noticed it a day before since I had it half implemented myself. :)

Quoted 8 lines

Hi Jeff,

Yes, it should be, and I wrote this cleanup_transaction where I should notice you earlier... Anyway, thanks for your effort.

The error handling part has lots of corner cases, so I just pick up a brute way to tear down the current transaction in order to make the FS RO.

Quoted 1 line

thanks, liubo

This afternoon I started running xfstests on a dm-linear mapped partition. Halfway through a sufficiently long test, I swap out the linear mapping to an error mapping. It still crashes, but somewhat less spectacularly. There are still a ton of BUG_ON's I need to eliminate as well as work out the usual I/O error-recovery issue of uninterruptible, unrecoverable writeback contexts and still-locked pages holding up exit. I'm pretty pleased with the results so far and am pretty optimistic.

-Jeff

Quoted 5 lines

-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 17 lines · Liu Bo

On 12/22/2011 10:59 AM, Jeff Mahoney wrote: Sorry I haven't responded to this yet. I started digging right in and I've started to have some good results. It turns out there's already a
btrfs_cleanup_transaction call that will tear down outstanding
transactions. It's not perfect and I've fixed a few bugs in there, but it saved me a bunch of effort. I just wished I noticed it a day before since I had it half implemented myself. :)

Quoted 8 lines

Hi Jeff,

Yes, it should be, and I wrote this cleanup_transaction where I should notice you earlier... Anyway, thanks for your effort.

The error handling part has lots of corner cases, so I just pick up a brute way to tear down the current transaction in order to make the FS RO.

Oh, and it's worked great. The brute force method is a good start and will address the most severe problems (and most cases) well. I've decided to ignore most cases of -ENOMEM for now. The biggest bug I ran into so far was calling mutex_lock while holding a spinlock. It was a quick fix.

The method I've generally used is to mark the transaction aborted and pass the error up as quickly as possible, cleaning up the local allocations and locks as I go. The transaction gets completed normally, returns an error, isn't committed, and then is destroyed (with others, potentially) when called from in btrfs_commit_transaction. Btrfs makes this super easy since we can just skip all the CoW writes.

Thanks!

-Jeff

Quoted 11 lines

Quoted 1 line

thanks, liubo

This afternoon I started running xfstests on a dm-linear mapped partition. Halfway through a sufficiently long test, I swap out the linear mapping to an error mapping. It still crashes, but somewhat less spectacularly. There are still a ton of BUG_ON's I need to eliminate as well as work out the usual I/O error-recovery issue of uninterruptible, unrecoverable writeback contexts and still-locked pages holding up exit. I'm pretty pleased with the results so far and am pretty optimistic.

Quoted 8 lines

-Jeff

Quoted 5 lines

-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 21 lines · Jeff Mahoney

Quoted 14 lines · Liu Bo

On 12/22/2011 10:59 AM, Jeff Mahoney wrote: Sorry I haven't responded to this yet. I started digging right in and I've started to have some good results. It turns out there's already a
btrfs_cleanup_transaction call that will tear down outstanding
transactions. It's not perfect and I've fixed a few bugs in there, but it saved me a bunch of effort. I just wished I noticed it a day before since I had it half implemented myself. :)

Quoted 6 lines

Hi Jeff,
Yes, it should be, and I wrote this cleanup_transaction where I should notice you earlier... Anyway, thanks for your effort. The error handling part has lots of corner cases, so I just pick up a brute way to tear down the current transaction in order to make the FS RO.

Oh, and it's worked great. The brute force method is a good start and will address the most severe problems (and most cases) well. I've decided to ignore most cases of -ENOMEM for now. The biggest bug I ran into so far was calling mutex_lock while holding a spinlock. It was a quick fix.

Quoted 7 lines

The method I've generally used is to mark the transaction aborted and pass the error up as quickly as possible, cleaning up the local allocations and locks as I go. The transaction gets completed normally, returns an error, isn't committed, and then is destroyed (with others, potentially) when called from in btrfs_commit_transaction. Btrfs makes this super easy since we can just skip all the CoW writes.

Now, just out of curiosity, would it be ok if I printed this when we ran out memory in deep call paths?

FAIL WHALE!

W W W
W W W W

              '.  W

.-""-._ \ \.--|
/ "-..__) .-'
| _ /
\'-.__, .__.,'
`'----'._\--'
VVVVVVVVVVVVVVVVVVVVV

Happy Holidays ;)

- -Jeff

Quoted 22 lines

Thanks!

-Jeff

Quoted 16 lines

Quoted 1 line

thanks, liubo

This afternoon I started running xfstests on a dm-linear mapped partition. Halfway through a sufficiently long test, I swap out the linear mapping to an error mapping. It still crashes, but somewhat less spectacularly. There are still a ton of BUG_ON's I need to eliminate as well as work out the usual I/O error-recovery issue of uninterruptible, unrecoverable writeback contexts and still-locked pages holding up exit. I'm pretty pleased with the results so far and am pretty optimistic. -Jeff

Quoted 5 lines

-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Quoted 3 lines

-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJO9A2/AAoJEB57S2MheeWyiNIP/3Z6NETIXskkp+OVKTiF/gaP bopj2dp92BlURFHEj5vJoESm4cUtQKTx9J/DB3yc7JDzc0UcRs9KCqGV9UpH6y9/ Zetzy3ZMsYyxvV5CZ50NGr+C1r5ULVGQ/UrPex/GT0bApcdBRMkFASLH8xkFl6dE dfRjir038GzjVX/Phy0VPm0mg8eg77aco11Xk2+Y1MdEhsEqI+cUQYgA8O9M7HWy 67Vv3KWxKC7PU6SYCPa0wGmQwTgs10GuKT9w+s7Ampy8iQhCgEuDo4dQxpRehQfp YwD/vlHwVATTAR2zMbRtI0BWa+ideBzcdQg1QrZxB3o026Z7ooy+/fTqS6MiUrXy mxGvb0g/BglK6Q86YQE77doIfJeUDLGoGQx2Zv1S9OzVwigo1a0LcP82P7yNnJBY oihql+FAYBXwjqiAQ+wUvo7wy0H+ltmQgWfUDf5wjDHquTRT1H0kE15Okc8MX8+T rmhp6vD1deX5Jz+JBIpCm94JhxUBPkBH2WksyA1jdLUOngHxRI0jmqz/5mPexV8e dChaq1rsjYs5Zbbv/jpaefnEw0kbZ0cqS7uDLVVoyjEqGnBpqjdwE86WYjxc4biM MkeSJ67Oof3ZGLWR0VQ+h4YnRjqAsMWsEd3jBLMo2krsr8ucc/UOzVDBVojDlGWJ Z2HunZuWJkNgcsBatVoS
=z1sd
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 25 lines · Jeff Mahoney

Quoted 17 lines · Liu Bo

On 12/22/2011 10:59 AM, Jeff Mahoney wrote: Sorry I haven't responded to this yet. I started digging right in and I've started to have some good results. It turns out there's already a
btrfs_cleanup_transaction call that will tear down outstanding
transactions. It's not perfect and I've fixed a few bugs in there, but it saved me a bunch of effort. I just wished I noticed it a day before since I had it half implemented myself. :)

Quoted 8 lines

Hi Jeff,

Yes, it should be, and I wrote this cleanup_transaction where I should notice you earlier... Anyway, thanks for your effort.

The error handling part has lots of corner cases, so I just pick up a brute way to tear down the current transaction in order to make the FS RO.

Oh, and it's worked great. The brute force method is a good start and will address the most severe problems (and most cases) well. I've decided to ignore most cases of -ENOMEM for now. The biggest bug I ran into so far was calling mutex_lock while holding a spinlock. It was a quick fix.

Quoted 7 lines

The method I've generally used is to mark the transaction aborted and pass the error up as quickly as possible, cleaning up the local allocations and locks as I go. The transaction gets completed normally, returns an error, isn't committed, and then is destroyed (with others, potentially) when called from in btrfs_commit_transaction. Btrfs makes this super easy since we can just skip all the CoW writes.

Now, just out of curiosity, would it be ok if I printed this when we ran out memory in deep call paths?

FAIL WHALE!

W W W
W W W W

              '.  W

.-""-._ \ \.--|
/ "-..__) .-'
| _ /
\'-.__, .__.,'
`'----'._\--'
VVVVVVVVVVVVVVVVVVVVV

Happy Holidays ;)