Skip to content

ext4: Fix transaction overflow due to revoke descriptors

Jan Kara linux-ext4 stable 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest0 of 19 patches 9 repliesIn review
  2. v219 patches 7 repliesSuperseded
  3. v115 patches 0 repliesSuperseded

Changes in v3 from the cover letter

* Fixed bug in revoke credit estimates for extent freeing in bigalloc filesystems
* Fixed bug in xattr code treating positive return of ext4_journal_ensure_credits() as error
* Fixed preexisting bug in ext4_evict_inode() where we reserved too few credits
* Added trace point to jbd2_journal_restart()
* Fix some kernel doc bugs
* Rebased on top of 5.4-rc1

Patches in v3 No applies recorded yet

None of the 19 patches the cover letter announces are in this archive.

Reviews trailers per patch

PatchTheodore Ts'o
SeriesR-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [PATCH 0/19 v3] ext4: Fix transaction overflow due to revoke descriptors

Hello,

Here is v3 of this series with couple more bugs fixed. Now all failed tests Ted higlighted pass for me.

Changes since v2:
* Fixed bug in revoke credit estimates for extent freeing in bigalloc filesystems
* Fixed bug in xattr code treating positive return of ext4_journal_ensure_credits() as error
* Fixed preexisting bug in ext4_evict_inode() where we reserved too few credits
* Added trace point to jbd2_journal_restart()
* Fix some kernel doc bugs
* Rebased on top of 5.4-rc1

Changes since v1:
* Reordered some patches to reduce code churn
* Computation in jbd2_revoke_descriptors_per_block() was too early - moved it to later when journal superblock is loaded and so the feature checking actually works.
* Made sure nobody outside JBD2 uses handle->h_buffer_credits since now it contains also credits for revoke descriptors and it was confusing come users.
* Updated cover letter with more details about reproducer

Original cover letter:

I've recently got a bug report where JBD2 assertion failed due to transaction commit running out of journal space. After closer inspection of the crash dump it seems that the problem is that there were too many journal descriptor blocks (more that max_transaction_size >> 5 + 32 we estimate in jbd2_log_space_left()) due to descriptor blocks with revoke records. In fact the estimate on the number of descriptor blocks looks pretty arbitrary and there can be much more descriptor blocks needed for revoke records. We need one revoke record for every metadata block freed. So in the worst case (1k blocksize, 64-bit journal feature enabled, checksumming enabled) we fit 125 revoke record in one descriptor block. In common cases its about 500 revoke records per descriptor block. Now when we free large directories or large file with data journalling enabled, we can have *lots* of blocks to revoke - with extent mapped files easily millions in a single transaction which can mean 10k descriptor blocks - clearly more than the estimate of 128 descriptor blocks per transaction ;)

This patch series aims at fixing the problem by accounting descriptor blocks into transaction credits and reserving appropriate amount of credits for revoke descriptors on transaction handle start. Similar to normal transaction credits, the filesystem has to provide estimate for the number of blocks it is going to revoke using the transaction handle so that credits for revoke descriptors can be reserved.

The series has survived fstests in couple configurations and also the stress test like:
Create filesystem with 1KB blocksize and journal size 32MB Mount the filesystem with -o nodelalloc
for (( i = 0; i < 4; i++ )); do
dd if=/dev/zero of=file$i bs=1M count=2048 conv=fsync
chattr +j file$i
done
for (( i = 0; i < 4; i++ )); do
rm file$i&
done

which reliably triggers the assertion failure in JBD2 on unpatched kernel.

Review and comments are welcome :).

Honza
Previous versions: