fs: Substitute bit-spinlocks for PREEMPT_RT and debugging
Subsystems (6)
v2 cover fs: Substitute bit-spinlocks for PREEMPT_RT and debugging
Bit spinlocks are problematic if PREEMPT_RT is enabled. They disable preemption, which is undesired for latency reasons and breaks when regular spinlocks are taken within the bit_spinlock locked region because regular spinlocks are converted to 'sleeping spinlocks' on RT.
Bit spinlocks are also not covered by lock debugging, e.g. lockdep. With the spinlock substitution in place, they can be exposed via a new config
switch: CONFIG_DEBUG_BIT_SPINLOCKS.
This series handles only buffer head and jbd2, but does not touch the hlist_bl bit-spinlock usage. See V1 for further details:
https://lkml.kernel.org/r/[email protected]
Changes vs. V1:
- Collected reviewed-by tags for the BH_Uptodate part
- Reworked the JBD2 part according to Jan's review:
- Convert state lock to a regular spinlock unconditionally
- Refactor jbd2_journal_put_journal_head() to be RT friendly by restricting the bit-spinlock held section to the minimum required operations.
That part is probably over-engineered, but I'm sure Jan will tell me sooner than later.
Thanks,
tglx
PATCH v2 1/7 locking/lockdep: Add Kconfig option for bit spinlocks
Some usage sites of bit spinlocks have a substitution with regular spinlocks which depends on CONFIG_PREEMPT_RT. But this substitution can also be used to expose these locks to the regular lock debugging infrastructure, e.g. lockdep.
As this increases the size of affected data structures significantly this is guarded by a separate Kconfig switch.
Note, that only the bit spinlocks which have a substitution implemented will be covered by this. All other bit spinlocks evade lock debugging as before.
--- lib/Kconfig.debug | 10 ++++++++++ 1 file changed, 10 insertions(+)
lib/Kconfig.debug +10 -0
@@ -1201,6 +1201,16 @@ config DEBUG_RWSEMS This debugging feature allows mismatched rw semaphore locks and unlocks to be detected and reported. +config DEBUG_BIT_SPINLOCKS + bool "Bit spinlock debugging" + depends on DEBUG_SPINLOCK + help + This debugging feature substitutes bit spinlocks in some use + cases, e.g. buffer head, zram, with with regular spinlocks so + these locks are exposed to lock debugging features. + + Not all bit spinlocks are covered by this. + config DEBUG_LOCK_ALLOC bool "Lock debugging: detect incorrect freeing of live locks" depends on DEBUG_KERNEL && LOCK_DEBUGGING_SUPPORT
PATCH v2 2/7 fs/buffer: Move BH_Uptodate_Lock locking into wrapper functions
In reply to Thomas GleixnerBit spinlocks are problematic if PREEMPT_RT is enabled, because they disable preemption, which is undesired for latency reasons and breaks when regular spinlocks are taken within the bit_spinlock locked region because regular spinlocks are converted to 'sleeping spinlocks' on RT. So RT replaces the bit spinlocks with regular spinlocks to avoid this problem.
To avoid ifdeffery at the source level, wrap all BH_Uptodate_Lock bitlock operations with inline functions, so the spinlock substitution can be done at one place.
Using regular spinlocks can also be enabled for lock debugging purposes so the lock operations become visible to lockdep.
---
V2: Collected Reviewed-by tag
--- fs/buffer.c | 20 ++++++-------------- fs/ext4/page-io.c | 6 ++---- fs/ntfs/aops.c | 10 +++------- include/linux/buffer_head.h | 16 ++++++++++++++++ 4 files changed, 27 insertions(+), 25 deletions(-)
fs/buffer.c +6 -14
@@ -275,8 +275,7 @@ static void end_buffer_async_read(struct* decide that the page is now completely done. */ first =page_buffers (page); -local_irq_save (flags); -bit_spin_lock (BH_Uptodate_Lock, &first->b_state); + flags =bh_uptodate_lock_irqsave (first);clear_buffer_async_read (bh);unlock_buffer (bh); tmp = bh;
@@ -289,8 +288,7 @@ static void end_buffer_async_read(struct } tmp = tmp->b_this_page; }while (tmp != bh); -bit_spin_unlock (BH_Uptodate_Lock, &first->b_state); -local_irq_restore (flags); +bh_uptodate_unlock_irqrestore (first, flags);/* * If none of the buffers had errors and they are all
@@ -302,9 +300,7 @@ static void end_buffer_async_read(structreturn ; still_busy: -bit_spin_unlock (BH_Uptodate_Lock, &first->b_state); -local_irq_restore (flags); -return ; +bh_uptodate_unlock_irqrestore (first, flags); }/*
@@ -331,8 +327,7 @@ void end_buffer_async_write(struct buffe } first =page_buffers (page); -local_irq_save (flags); -bit_spin_lock (BH_Uptodate_Lock, &first->b_state); + flags =bh_uptodate_lock_irqsave (first);clear_buffer_async_write (bh);unlock_buffer (bh);
@@ -344,15 +339,12 @@ void end_buffer_async_write(struct buffe } tmp = tmp->b_this_page; } -bit_spin_unlock (BH_Uptodate_Lock, &first->b_state); -local_irq_restore (flags); +bh_uptodate_unlock_irqrestore (first, flags);end_page_writeback (page);return ; still_busy: -bit_spin_unlock (BH_Uptodate_Lock, &first->b_state); -local_irq_restore (flags); -return ; +bh_uptodate_unlock_irqrestore (first, flags); }EXPORT_SYMBOL (end_buffer_async_write);
fs/ext4/page-io.c +2 -4
@@ -90,8 +90,7 @@ static void ext4_finish_bio(struct bio ** We check all buffers in the page under BH_Uptodate_Lock * to avoid races with other end io clearing async_write flags */ -local_irq_save (flags); -bit_spin_lock (BH_Uptodate_Lock, &head->b_state); + flags =bh_uptodate_lock_irqsave (head); do { if (bh_offset(bh) < bio_start || bh_offset(bh) + bh->b_size > bio_end) {
@@ -103,8 +102,7 @@ static void ext4_finish_bio(struct bio *if (bio->bi_status)buffer_io_error (bh); }while ((bh = bh->b_this_page) != head); -bit_spin_unlock (BH_Uptodate_Lock, &head->b_state); -local_irq_restore (flags); +bh_uptodate_unlock_irqrestore (head, flags);if (!under_io) {fscrypt_free_bounce_page (bounce_page);end_page_writeback (page);
fs/ntfs/aops.c +3 -7
@@ -92,8 +92,7 @@ static void ntfs_end_buffer_async_read(s "0x%llx.", (unsigned long long)bh->b_blocknr); } first =page_buffers (page); -local_irq_save (flags); -bit_spin_lock (BH_Uptodate_Lock, &first->b_state); + flags =bh_uptodate_lock_irqsave (first);clear_buffer_async_read (bh);unlock_buffer (bh); tmp = bh;
@@ -108,8 +107,7 @@ static void ntfs_end_buffer_async_read(s } tmp = tmp->b_this_page; }while (tmp != bh); -bit_spin_unlock (BH_Uptodate_Lock, &first->b_state); -local_irq_restore (flags); +bh_uptodate_unlock_irqrestore (first, flags);/* * If none of the buffers had errors then we can set the page uptodate, * but we first have to perform the post read mst fixups, if the
@@ -142,9 +140,7 @@ static void ntfs_end_buffer_async_read(sunlock_page (page);return ; still_busy: -bit_spin_unlock (BH_Uptodate_Lock, &first->b_state); -local_irq_restore (flags); -return ; +bh_uptodate_unlock_irqrestore (first, flags); }/**
include/linux/buffer_head.h +16 -0
@@ -78,6 +78,22 @@ struct buffer_head {atomic_t b_count;/* users using this buffer_head */ }; +static inline unsigned long bh_uptodate_lock_irqsave (struct buffer_head *bh) +{ +unsigned long flags; + +local_irq_save (flags); +bit_spin_lock (BH_Uptodate_Lock, &bh->b_state); +return flags; +} + +static inline void +bh_uptodate_unlock_irqrestore (struct buffer_head *bh,unsigned long flags) +{ +bit_spin_unlock (BH_Uptodate_Lock, &bh->b_state); +local_irq_restore (flags); +} +/* * macro tricks to expand the set_buffer_foo(), clear_buffer_foo() * and buffer_foo() functions.
PATCH v2 3/7 fs/buffer: Substitute BH_Uptodate_Lock for RT and bit spinlock debugging
In reply to Thomas GleixnerBit spinlocks are problematic if PREEMPT_RT is enabled. They disable preemption, which is undesired for latency reasons and breaks when regular spinlocks are taken within the bit_spinlock locked region because regular spinlocks are converted to 'sleeping spinlocks' on RT.
Substitute the BH_Uptodate_Lock bit spinlock with a regular spinlock for PREEMPT_RT enabled kernels.
Bit spinlocks are also not covered by lock debugging, e.g. lockdep. With the spinlock substitution in place, they can be exposed via CONFIG_DEBUG_BIT_SPINLOCKS.
---
V2: Collected Reviewed-by tag
--- fs/buffer.c | 1 + include/linux/buffer_head.h | 31 +++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)
fs/buffer.c +1 -0
@@ -3360,6 +3360,7 @@ struct buffer_head *alloc_buffer_head(gfstruct buffer_head *ret =kmem_cache_zalloc (bh_cachep, gfp_flags);if (ret) {INIT_LIST_HEAD (&ret->b_assoc_buffers); +buffer_head_init_locks (ret);preempt_disable ();__this_cpu_inc (bh_accounting.nr);recalc_bh_state ();
include/linux/buffer_head.h +31 -0
@@ -76,8 +76,35 @@ struct buffer_head {struct address_space *b_assoc_map;/* mapping this buffer is associated with */ atomic_t b_count;/* users using this buffer_head */ + +#if defined(CONFIG_PREEMPT_RT ) || defined(CONFIG_DEBUG_BIT_SPINLOCKS ) +spinlock_t b_uptodate_lock; +#endif }; +#if defined(CONFIG_PREEMPT_RT ) || defined(CONFIG_DEBUG_BIT_SPINLOCKS ) + +static inline unsigned long bh_uptodate_lock_irqsave (struct buffer_head *bh) +{ +unsigned long flags; + +spin_lock_irqsave (&bh->b_uptodate_lock, flags); +return flags; +} + +static inline void +bh_uptodate_unlock_irqrestore (struct buffer_head *bh,unsigned long flags) +{ +spin_unlock_irqrestore (&bh->b_uptodate_lock, flags); +} + +static inline void buffer_head_init_locks (struct buffer_head *bh) +{ +spin_lock_init (&bh->b_uptodate_lock); +} + +#else /* PREEMPT_RT || DEBUG_BIT_SPINLOCKS */ +static inline unsigned long bh_uptodate_lock_irqsave (struct buffer_head *bh) {unsigned long flags;
@@ -94,6 +121,10 @@ bh_uptodate_unlock_irqrestore(struct buf local_irq_restore(flags); } +static inline void buffer_head_init_locks (struct buffer_head *bh) { } + +#endif /* !PREEMPT_RT && !DEBUG_BIT_SPINLOCKS */ +/* * macro tricks to expand the set_buffer_foo(), clear_buffer_foo() * and buffer_foo() functions.
PATCH v2 4/7 fs/jbd2: Remove jbd_trylock_bh_state()
In reply to Thomas GleixnerNo users.
--- include/linux/jbd2.h | 5 ----- 1 file changed, 5 deletions(-)
include/linux/jbd2.h +0 -5
@@ -347,11 +347,6 @@ static inline void jbd_lock_bh_state(strbit_spin_lock (BH_State, &bh->b_state); } -static inline int jbd_trylock_bh_state (struct buffer_head *bh) -{ -return bit_spin_trylock (BH_State, &bh->b_state); -} -static inline int jbd_is_locked_bh_state (struct buffer_head *bh) {return bit_spin_is_locked (BH_State, &bh->b_state);
Re: [patch V2 4/7] fs/jbd2: Remove jbd_trylock_bh_state()
Quoted 6 lines · Thomas Gleixner
No users.
Signed-off-by Thomas Gleixner <[email protected]>Cc "Theodore Ts'o" <[email protected]>Cc Jan Kara <[email protected]>
Right.
Honza
Quoted 20 lines
--- include/linux/jbd2.h | 5 ----- 1 file changed, 5 deletions(-)include/linux/jbd2.h +0
-5@@ -347,11 +347,6 @@ static inline void jbd_lock_bh_state(strbit_spin_lock (BH_State, &bh->b_state); } -static inline int jbd_trylock_bh_state (struct buffer_head *bh) -{ -return bit_spin_trylock (BH_State, &bh->b_state); -} -static inline int jbd_is_locked_bh_state (struct buffer_head *bh) {return bit_spin_is_locked (BH_State, &bh->b_state);
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH v2 5/7 fs/jbd2: Simplify journal_unmap_buffer()
In reply to Thomas Gleixnerjournal_unmap_buffer() checks first whether the buffer head is a journal. If so it takes locks and then invokes jbd2_journal_grab_journal_head() followed by another check whether this is journal head buffer.
The double checking is pointless.
Replace the initial check with jbd2_journal_grab_journal_head() which alredy checks whether the buffer head is actually a journal.
Allows also early access to the journal head pointer for the upcoming conversion of state lock to a regular spinlock.
---
V2: New patch
--- fs/jbd2/transaction.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
fs/jbd2/transaction.c +2 -6
@@ -2196,7 +2196,8 @@ static int journal_unmap_buffer(journal_* holding the page lock. --sct */ -if (!buffer_jbd (bh)) + jh =jbd2_journal_grab_journal_head (bh); +if (!jh) goto zap_buffer_unlocked;/* OK, we have data buffer in journaled mode */
@@ -2204,10 +2205,6 @@ static int journal_unmap_buffer(journal_jbd_lock_bh_state (bh);spin_lock (&journal->j_list_lock); - jh =jbd2_journal_grab_journal_head (bh); -if (!jh) - goto zap_buffer_no_jh; -/* * We cannot remove the buffer from checkpoint lists until the * transaction adding inode to orphan list (let's call it T)
@@ -2329,7 +2326,6 @@ static int journal_unmap_buffer(journal_*/ jh->b_modified =0 ;jbd2_journal_put_journal_head (jh); -zap_buffer_no_jh:spin_unlock (&journal->j_list_lock);jbd_unlock_bh_state (bh);write_unlock (&journal->j_state_lock);
Re: [patch V2 5/7] fs/jbd2: Simplify journal_unmap_buffer()
Quoted 9 lines · Thomas Gleixner
journal_unmap_buffer()checks first whether the buffer head is a journal. If so it takes locks and then invokesjbd2_journal_grab_journal_head()followed by another check whether this is journal head buffer.The double checking is pointless.
Replace the initial check with
jbd2_journal_grab_journal_head()which alredy checks whether the buffer head is actually a journal.
Quoted 7 lines
Allows also early access to the journal head pointer for the upcoming conversion of state lock to a regular spinlock.
Signed-off-by Thomas Gleixner <[email protected]>Cc "Theodore Ts'o" <[email protected]>Cc Jan Kara <[email protected]>
Nice simplification. You can add:
Honza
Quoted 39 lines
---
V2: New patch--- fs/jbd2/transaction.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)fs/jbd2/transaction.c +2
-6@@ -2196,7 +2196,8 @@ static int journal_unmap_buffer(journal_* holding the page lock. --sct */ -if (!buffer_jbd (bh)) + jh =jbd2_journal_grab_journal_head (bh); +if (!jh) goto zap_buffer_unlocked;/* OK, we have data buffer in journaled mode */ @@ -2204,10 +2205,6 @@ static int journal_unmap_buffer(journal_jbd_lock_bh_state (bh);spin_lock (&journal->j_list_lock); - jh =jbd2_journal_grab_journal_head (bh); -if (!jh) - goto zap_buffer_no_jh; -/* * We cannot remove the buffer from checkpoint lists until the * transaction adding inode to orphan list (let's call it T) @@ -2329,7 +2326,6 @@ static int journal_unmap_buffer(journal_*/ jh->b_modified =0 ;jbd2_journal_put_journal_head (jh); -zap_buffer_no_jh:spin_unlock (&journal->j_list_lock);jbd_unlock_bh_state (bh);write_unlock (&journal->j_state_lock);
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH v2 6/7 fs/jbd2: Make state lock a spinlock
In reply to Thomas GleixnerBit-spinlocks are problematic on PREEMPT_RT if functions which might sleep on RT, e.g. spin_lock(), alloc/free(), are invoked inside the lock held region because bit spinlocks disable preemption even on RT.
A first attempt was to replace state lock with a spinlock placed in struct buffer_head and make the locking conditional on PREEMPT_RT and DEBUG_BIT_SPINLOCKS.
Jan pointed out that there is a 4 byte hole in struct journal_head where a regular spinlock fits in and he would not object to convert the state lock to a spinlock unconditionally.
Aside of solving the RT problem, this also gains lockdep coverage for the journal head state lock (bit-spinlocks are not covered by lockdep as it's hard to fit a lockdep map into a single bit).
The trivial change would have been to convert the jbd_*lock_bh_state() inlines, but that comes with the downside that these functions take a buffer head pointer which needs to be converted to a journal head pointer which adds another level of indirection.
As almost all functions which use this lock have a journal head pointer readily available, it makes more sense to remove the lock helper inlines and write out spin_*lock() at all call sites.
Fixup all locking comments as well.
---
V2: New patch
--- fs/jbd2/commit.c | 8 ++-- fs/jbd2/journal.c | 10 +++-- fs/jbd2/transaction.c | 83 +++++++++++++++++++++---------------------- fs/ocfs2/suballoc.c | 19 +++++---- include/linux/jbd2.h | 20 +--------- include/linux/journal-head.h | 19 ++++++--- 6 files changed, 77 insertions(+), 82 deletions(-)
fs/jbd2/commit.c +4 -4
@@ -482,10 +482,10 @@ void jbd2_journal_commit_transaction(jouif (jh->b_committed_data) {struct buffer_head *bh =jh2bh (jh); -jbd_lock_bh_state (bh); +spin_lock (&jh->state_lock);jbd2_free (jh->b_committed_data, bh->b_size); jh->b_committed_data =NULL ; -jbd_unlock_bh_state (bh); +spin_unlock (&jh->state_lock); }jbd2_journal_refile_buffer (journal, jh); }
@@ -927,7 +927,7 @@ void jbd2_journal_commit_transaction(jou* done with it. */ get_bh (bh); -jbd_lock_bh_state (bh); +spin_lock (&jh->state_lock);J_ASSERT_JH (jh, jh->b_transaction == commit_transaction);/*
@@ -1023,7 +1023,7 @@ void jbd2_journal_commit_transaction(jou }JBUFFER_TRACE (jh,"refile or unfile buffer" );__jbd2_journal_refile_buffer (jh); -jbd_unlock_bh_state (bh); +spin_unlock (&jh->state_lock);if (try_to_free)release_buffer_page (bh);/* Drops bh reference */ else
fs/jbd2/journal.c +6 -4
@@ -365,7 +365,7 @@ int jbd2_journal_write_metadata_buffer(t/* keep subsequent assertions sane */ atomic_set (&new_bh->b_count,1 ); -jbd_lock_bh_state (bh_in); +spin_lock (&jh_in->state_lock); repeat:/* * If a new transaction has already done a buffer copy-out, then
@@ -407,13 +407,13 @@ int jbd2_journal_write_metadata_buffer(tif (need_copy_out && !done_copy_out) {char *tmp; -jbd_unlock_bh_state (bh_in); +spin_unlock (&jh_in->state_lock); tmp =jbd2_alloc (bh_in->b_size,GFP_NOFS );if (!tmp) {brelse (new_bh);return -ENOMEM ; } -jbd_lock_bh_state (bh_in); +spin_lock (&jh_in->state_lock);if (jh_in->b_frozen_data) {jbd2_free (tmp, bh_in->b_size); goto repeat;
@@ -466,7 +466,7 @@ int jbd2_journal_write_metadata_buffer(t__jbd2_journal_file_buffer (jh_in, transaction, BJ_Shadow);spin_unlock (&journal->j_list_lock);set_buffer_shadow (bh_in); -jbd_unlock_bh_state (bh_in); +spin_unlock (&jh_in->state_lock);return do_escape | (done_copy_out <<1 ); }
@@ -2412,6 +2412,8 @@ static struct journal_head *journal_allo ret =kmem_cache_zalloc (jbd2_journal_head_cache,GFP_NOFS | __GFP_NOFAIL); } +if (ret) +spin_lock_init (&ret->state_lock);return ret; }
fs/jbd2/transaction.c +42 -41
306 lines not on this page · raw message
fs/ocfs2/suballoc.c +11 -8
@@ -1252,6 +1252,7 @@ static int ocfs2_test_bg_bit_allocatable int nr) {struct ocfs2_group_desc *bg = (struct ocfs2_group_desc *) bg_bh->b_data; +struct journal_head *jh;int ret;if (ocfs2_test_bit (nr, (unsigned long *)bg->bg_bitmap))
@@ -1260,13 +1261,14 @@ static int ocfs2_test_bg_bit_allocatableif (!buffer_jbd (bg_bh))return 1 ; -jbd_lock_bh_state (bg_bh); - bg = (struct ocfs2_group_desc *)bh2jh (bg_bh)->b_committed_data; + jh =bh2jh (bg_bh); +spin_lock (&jh->state_lock); + bg = (struct ocfs2_group_desc *) jh->b_committed_data;if (bg) ret = !ocfs2_test_bit (nr, (unsigned long *)bg->bg_bitmap);else ret =1 ; -jbd_unlock_bh_state (bg_bh); +spin_unlock (&jh->state_lock);return ret; }
@@ -2387,6 +2389,7 @@ static int ocfs2_block_group_clear_bits(int status;unsigned int tmp;struct ocfs2_group_desc *undo_bg =NULL ; +struct journal_head *jh;/* The caller got this descriptor from * ocfs2_read_group_descriptor(). Any corruption is a code bug. */
@@ -2405,10 +2408,10 @@ static int ocfs2_block_group_clear_bits( goto bail; } + jh =bh2jh (group_bh);if (undo_fn) { -jbd_lock_bh_state (group_bh); - undo_bg = (struct ocfs2_group_desc *) -bh2jh (group_bh)->b_committed_data; +spin_lock (&jh->state_lock); + undo_bg = (struct ocfs2_group_desc *) jh->b_committed_data;BUG_ON (!undo_bg); }
@@ -2423,7 +2426,7 @@ static int ocfs2_block_group_clear_bits(le16_add_cpu (&bg->bg_free_bits_count, num_bits);if (le16_to_cpu (bg->bg_free_bits_count) >le16_to_cpu (bg->bg_bits)) {if (undo_fn) -jbd_unlock_bh_state (group_bh); +spin_unlock (&jh->state_lock); return ocfs2_error(alloc_inode->i_sb, "Group descriptor # %llu has bit count %u but claims %u are freed. num_bits %d\n", (unsigned long long)le64_to_cpu(bg->bg_blkno), le16_to_cpu(bg->bg_bits),
@@ -2432,7 +2435,7 @@ static int ocfs2_block_group_clear_bits( }if (undo_fn) -jbd_unlock_bh_state (group_bh); +spin_unlock (&jh->state_lock);ocfs2_journal_dirty (handle, group_bh); bail:
include/linux/jbd2.h +2 -18
@@ -313,7 +313,6 @@ enum jbd_state_bits { BH_Revoked,/* Has been revoked from the log */ BH_RevokeValid,/* Revoked flag is valid */ BH_JBDDirty,/* Is dirty but journaled */ - BH_State,/* Pins most journal_head state */ BH_JournalHead,/* Pins bh->b_private and jh->b_bh */ BH_Shadow,/* IO on shadow buffer is running */ BH_Verified, /* Metadata block has been verified ok */
@@ -342,21 +341,6 @@ static inline struct journal_head *bh2jhreturn bh->b_private; } -static inline void jbd_lock_bh_state (struct buffer_head *bh) -{ -bit_spin_lock (BH_State, &bh->b_state); -} - -static inline int jbd_is_locked_bh_state (struct buffer_head *bh) -{ -return bit_spin_is_locked (BH_State, &bh->b_state); -} - -static inline void jbd_unlock_bh_state (struct buffer_head *bh) -{ -bit_spin_unlock (BH_State, &bh->b_state); -} -static inline void jbd_lock_bh_journal_head (struct buffer_head *bh) {bit_spin_lock (BH_JournalHead, &bh->b_state);
@@ -551,9 +535,9 @@ struct transaction_chp_stats_s {* ->jbd_lock_bh_journal_head() (This is "innermost") * * j_state_lock -* ->jbd_lock_bh_state() +* ->jh->state_lock * -* jbd_lock_bh_state() +* jh->state_lock * ->j_list_lock * * j_state_lock
include/linux/journal-head.h +12 -7
@@ -24,13 +24,18 @@ struct journal_head {struct buffer_head *b_bh;/* +* Protect the buffer head state +*/ +spinlock_t state_lock; + +/* * Reference count - see description in journal.c * [jbd_lock_bh_journal_head()] */ int b_jcount;/* -* Journalling list for this buffer [jbd_lock_bh_state()] +* Journalling list for this buffer [jh->state_lock] * NOTE: We *cannot* combine this with b_modified into a bitfield * as gcc would then (which the C standard allows but which is * very unuseful) make 64-bit accesses to the bitfield and clobber
@@ -41,20 +46,20 @@ struct journal_head {/* * This flag signals the buffer has been modified by * the currently running transaction -* [jbd_lock_bh_state()] +* [jh->state_lock] */ unsigned b_modified;/* * Copy of the buffer data frozen for writing to the log. -* [jbd_lock_bh_state()] +* [jh->state_lock] */ char *b_frozen_data;/* * Pointer to a saved copy of the buffer containing no uncommitted * deallocation references, so that allocations can avoid overwriting -* uncommitted deletes. [jbd_lock_bh_state()] +* uncommitted deletes. [jh->state_lock] */ char *b_committed_data;
@@ -63,7 +68,7 @@ struct journal_head {* metadata: either the running transaction or the committing * transaction (if there is one). Only applies to buffers on a * transaction's data or metadata journaling list. -* [j_list_lock] [jbd_lock_bh_state()] +* [j_list_lock] [jh->state_lock] * Either of these locks is enough for reading, both are needed for * changes. */
@@ -73,13 +78,13 @@ struct journal_head {* Pointer to the running compound transaction which is currently * modifying the buffer's metadata, if there was already a transaction * committing it when the new transaction touched it. -* [t_list_lock] [jbd_lock_bh_state()] +* [t_list_lock] [jh->state_lock] */ transaction_t *b_next_transaction;/* * Doubly-linked list of buffers on a transaction's data, metadata or -* forget queue. [t_list_lock] [jbd_lock_bh_state()] +* forget queue. [t_list_lock] [jh->state_lock] */ struct journal_head *b_tnext, *b_tprev;
Re: [patch V2 6/7] fs/jbd2: Make state lock a spinlock
Quoted 21 lines · Thomas Gleixner
fs/jbd2/transaction.c +2
-2@@ -1931,7 +1932,7 @@ static void __jbd2_journal_temp_unlink_btransaction_t *transaction;struct buffer_head *bh =jh2bh (jh); -J_ASSERT_JH (jh,jbd_is_locked_bh_state (bh)); +assert_spin_locked (&jh->state_lock); transaction = jh->b_transaction;if (transaction)assert_spin_locked (&transaction->t_journal->j_list_lock);@@ -2415,7 +2416,7 @@ void __jbd2_journal_file_buffer(struct jint was_dirty =0 ;struct buffer_head *bh =jh2bh (jh); -J_ASSERT_JH (jh,jbd_is_locked_bh_state (bh)); +assert_spin_locked (&jh->state_lock);assert_spin_locked (&transaction->t_journal->j_list_lock);J_ASSERT_JH (jh, jh->b_jlist < BJ_Types);
fs/jbd2/transaction.c void __jbd2_journal_refile_buffer(struct · line 2507
Quoted 8 lines
fs/jbd2/transaction.c +1
-1@@ -2500,7 +2501,7 @@ void __jbd2_journal_refile_buffer(structint was_dirty, jlist;struct buffer_head *bh =jh2bh (jh); -J_ASSERT_JH (jh,jbd_is_locked_bh_state (bh)); +assert_spin_locked (&jh->state_lock);if (jh->b_transaction)assert_spin_locked (&jh->b_transaction->t_journal->j_list_lock);
Do those want to be:
lockdep_assert_held(&jh->state_lock);
instead? The difference is of course that lockdep_assert_held() requires the current context to hold the lock, where assert_*_locked() merely checks _someone_ holds it.
Jan Kara
reviewerRe: [patch V2 6/7] fs/jbd2: Make state lock a spinlock
Quoted 34 lines · Peter Zijlstra
Quoted 21 lines · Thomas Gleixner
fs/jbd2/transaction.c +2
-2@@ -1931,7 +1932,7 @@ static void __jbd2_journal_temp_unlink_btransaction_t *transaction;struct buffer_head *bh =jh2bh (jh); -J_ASSERT_JH (jh,jbd_is_locked_bh_state (bh)); +assert_spin_locked (&jh->state_lock); transaction = jh->b_transaction;if (transaction)assert_spin_locked (&transaction->t_journal->j_list_lock);@@ -2415,7 +2416,7 @@ void __jbd2_journal_file_buffer(struct jint was_dirty =0 ;struct buffer_head *bh =jh2bh (jh); -J_ASSERT_JH (jh,jbd_is_locked_bh_state (bh)); +assert_spin_locked (&jh->state_lock);assert_spin_locked (&transaction->t_journal->j_list_lock);J_ASSERT_JH (jh, jh->b_jlist < BJ_Types);fs/jbd2/transaction.c void __jbd2_journal_refile_buffer(struct · line 2507
Quoted 8 lines
fs/jbd2/transaction.c +1
-1@@ -2500,7 +2501,7 @@ void __jbd2_journal_refile_buffer(structint was_dirty, jlist;struct buffer_head *bh =jh2bh (jh); -J_ASSERT_JH (jh,jbd_is_locked_bh_state (bh)); +assert_spin_locked (&jh->state_lock);if (jh->b_transaction)assert_spin_locked (&jh->b_transaction->t_journal->j_list_lock);Do those want to be:
Quoted 5 lines
lockdep_assert_held(&jh->state_lock);instead? The difference is of course that
lockdep_assert_held()requires the current context to hold the lock, where assert_*_locked() merely checks _someone_ holds it.
Yeah, jbd2 doesn't play any weird locking tricks so lockdep_assert_held() is fine. I'll replace those when I'm updating the patch.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
reviewerRe: [patch V2 6/7] fs/jbd2: Make state lock a spinlock
In reply to Thomas GleixnerQuoted 27 lines · Thomas Gleixner
Bit-spinlocks are problematic on
PREEMPT_RTif functions which might sleep on RT, e.g.spin_lock(), alloc/free(), are invoked inside the lock held region because bit spinlocks disable preemption even on RT.A first attempt was to replace state lock with a spinlock placed in struct
buffer_headand make the locking conditional onPREEMPT_RTandDEBUG_BIT_SPINLOCKS.Jan pointed out that there is a 4 byte hole in struct
journal_headwhere a regular spinlock fits in and he would not object to convert the state lock to a spinlock unconditionally.Aside of solving the RT problem, this also gains lockdep coverage for the journal head state lock (bit-spinlocks are not covered by lockdep as it's hard to fit a lockdep map into a single bit).
The trivial change would have been to convert the jbd_*lock_bh_state() inlines, but that comes with the downside that these functions take a buffer head pointer which needs to be converted to a journal head pointer which adds another level of indirection.
As almost all functions which use this lock have a journal head pointer readily available, it makes more sense to remove the lock helper inlines and write out spin_*lock() at all call sites.
Fixup all locking comments as well.
Quoted 8 lines
Suggested-by Jan Kara <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>Cc "Theodore Ts'o" <[email protected]>Cc Mark Fasheh <[email protected]>Cc Joseph Qi <[email protected]>Cc Joel Becker <[email protected]>Cc Jan Kara <[email protected]>
Just a heads up that I didn't miss this patch. Just it has some bugs and I figured that rather than explaining to you subtleties of jh lifetime it is easier to fix up the problems myself since you're probably not keen on becoming jbd2 developer ;)... which was more complex than I thought so I'm not completely done yet. Hopefuly tomorrow.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [patch V2 6/7] fs/jbd2: Make state lock a spinlock
Quoted 11 lines · Jan Kara
Quoted 4 lines · Thomas Gleixner
As almost all functions which use this lock have a journal head pointer readily available, it makes more sense to remove the lock helper inlines and write out spin_*lock() at all call sites.
Just a heads up that I didn't miss this patch. Just it has some bugs and I figured that rather than explaining to you subtleties of jh lifetime it is easier to fix up the problems myself since you're probably not keen on becoming jbd2 developer ;)... which was more complex than I thought so I'm not completely done yet. Hopefuly tomorrow.
I'm curious where I was too naive :)
Thanks for having a look!
tglx
Jan Kara
reviewerRe: [patch V2 6/7] fs/jbd2: Make state lock a spinlock
Quoted 7 lines · Thomas Gleixner
Quoted 6 lines · Jan Kara
Quoted 4 lines · Thomas Gleixner
As almost all functions which use this lock have a journal head pointer readily available, it makes more sense to remove the lock helper inlines and write out spin_*lock() at all call sites.
Quoted 7 lines
Quoted 5 lines
Just a heads up that I didn't miss this patch. Just it has some bugs and I figured that rather than explaining to you subtleties of jh lifetime it is easier to fix up the problems myself since you're probably not keen on becoming jbd2 developer ;)... which was more complex than I thought so I'm not completely done yet. Hopefuly tomorrow.
I'm curious where I was too naive :)
Well, the most obvious where places where the result ended up being like
jbd2_journal_put_journal_head(jh); spin_unlock(&jh->state_lock);
That's possible use-after-free.
But there were also other more subtle places where jbd2_journal_put_journal_head() was not directly visible as it was hidden inside journal list handling functions such as __jbd2_journal_refile_buffer() or so. And these needed some more work.
Anyway, I'm now done fixing up the patch, doing some xfstests runs to verify things didn't break in any obvious way...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [patch V2 6/7] fs/jbd2: Make state lock a spinlock
Quoted 14 lines · Jan Kara
Quoted 13 lines · Thomas Gleixner
Quoted 11 lines · Jan Kara
Quoted 4 lines · Thomas Gleixner
As almost all functions which use this lock have a journal head pointer readily available, it makes more sense to remove the lock helper inlines and write out spin_*lock() at all call sites.
Just a heads up that I didn't miss this patch. Just it has some bugs and I figured that rather than explaining to you subtleties of jh lifetime it is easier to fix up the problems myself since you're probably not keen on becoming jbd2 developer ;)... which was more complex than I thought so I'm not completely done yet. Hopefuly tomorrow.
Quoted 8 lines
Quoted 1 line
I'm curious where I was too naive :)
Well, the most obvious where places where the result ended up being like
jbd2_journal_put_journal_head(jh); spin_unlock(&jh->state_lock);That's possible use-after-free.
Duh yes.
Quoted 7 lines
But there were also other more subtle places where
jbd2_journal_put_journal_head()was not directly visible as it was hidden inside journal list handling functions such as__jbd2_journal_refile_buffer()or so. And these needed some more work.Anyway, I'm now done fixing up the patch, doing some xfstests runs to verify things didn't break in any obvious way...
Very appreciated.
Thanks,
tglx
PATCH v2 7/7 fs/jbd2: Free journal head outside of locked region
In reply to Thomas GleixnerOn PREEMPT_RT bit-spinlocks have the same semantics as on PREEMPT_RT=n, i.e. they disable preemption. That means functions which are not safe to be called in preempt disabled context on RT trigger a might_sleep() assert.
The journal head bit spinlock is mostly held for short code sequences with trivial RT safe functionality, except for one place:
jbd2_journal_put_journal_head() invokes __journal_remove_journal_head() with the journal head bit spinlock held. __journal_remove_journal_head() invokes kmem_cache_free() which must not be called with preemption disabled on RT.
Jan suggested to rework the removal function so the actual free happens outside the bit-spinlocked region.
Split it into two parts:
- Do the sanity checks and the buffer head detach under the lock
- Do the actual free after dropping the lock
There is error case handling in the free part which needs to dereference the b_size field of the now detached buffer head. Due to paranoia (caused by ignorance) the size is retrieved in the detach function and handed into the free function. Might be over-engineered, but better safe than sorry.
This makes the journal head bit-spinlock usage RT compliant and also avoids nested locking which is not covered by lockdep.
---
V2: New patch
--- fs/jbd2/journal.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
fs/jbd2/journal.c +20 -8
@@ -2521,9 +2521,10 @@ struct journal_head *jbd2_journal_grab_jreturn jh; } -static void __journal_remove_journal_head (struct buffer_head *bh) +static size_t __journal_remove_journal_head (struct buffer_head *bh) {struct journal_head *jh =bh2jh (bh); +size_t b_size =READ_ONCE (bh->b_size);J_ASSERT_JH (jh, jh->b_jcount >=0 );J_ASSERT_JH (jh, jh->b_transaction ==NULL );
@@ -2533,17 +2534,25 @@ static void __journal_remove_journal_heaJ_ASSERT_BH (bh,buffer_jbd (bh));J_ASSERT_BH (bh,jh2bh (jh) == bh);BUFFER_TRACE (bh,"remove journal_head" ); + +/* Unlink before dropping the lock */ + bh->b_private =NULL ; + jh->b_bh =NULL ;/* debug, really */ +clear_buffer_jbd (bh); + +return b_size; +} + +static void journal_release_journal_head (struct journal_head *jh,size_t b_size) +{if (jh->b_frozen_data) {printk (KERN_WARNING "%s: freeing b_frozen_data\n" , __func__); -jbd2_free (jh->b_frozen_data, bh->b_size); +jbd2_free (jh->b_frozen_data, b_size); }if (jh->b_committed_data) {printk (KERN_WARNING "%s: freeing b_committed_data\n" , __func__); -jbd2_free (jh->b_committed_data, bh->b_size); +jbd2_free (jh->b_committed_data, b_size); } - bh->b_private =NULL ; - jh->b_bh =NULL ;/* debug, really */ -clear_buffer_jbd (bh);journal_free_journal_head (jh); }
@@ -2559,11 +2568,14 @@ void jbd2_journal_put_journal_head(strucJ_ASSERT_JH (jh, jh->b_jcount >0 ); --jh->b_jcount;if (!jh->b_jcount) { -__journal_remove_journal_head (bh); +size_t b_size =__journal_remove_journal_head (bh); +jbd_unlock_bh_journal_head (bh); +journal_release_journal_head (jh, b_size);__brelse (bh); - }else + }else {jbd_unlock_bh_journal_head (bh); + } }/*
Jan Kara
reviewerRe: [patch V2 7/7] fs/jbd2: Free journal head outside of locked region
Quoted 26 lines · Thomas Gleixner
On
PREEMPT_RTbit-spinlocks have the same semantics as on PREEMPT_RT=n, i.e. they disable preemption. That means functions which are not safe to be called in preempt disabled context on RT trigger amight_sleep()assert.The journal head bit spinlock is mostly held for short code sequences with trivial RT safe functionality, except for one place:
jbd2_journal_put_journal_head()invokes__journal_remove_journal_head()with the journal head bit spinlock held.__journal_remove_journal_head()invokeskmem_cache_free()which must not be called with preemption disabled on RT.Jan suggested to rework the removal function so the actual free happens outside the bit-spinlocked region.
Split it into two parts:
- Do the sanity checks and the buffer head detach under the lock
- Do the actual free after dropping the lock
There is error case handling in the free part which needs to dereference the
b_sizefield of the now detached buffer head. Due to paranoia (caused by ignorance) the size is retrieved in the detach function and handed into the free function. Might be over-engineered, but better safe than sorry.
Quoted 8 lines
This makes the journal head bit-spinlock usage RT compliant and also avoids nested locking which is not covered by lockdep.
Suggested-by Jan Kara <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>Cc "Theodore Ts'o" <[email protected]>Cc Jan Kara <[email protected]>
Looks mostly good. Just a small suggestion for simplification below:
fs/jbd2/journal.c void jbd2_journal_put_journal_head(struc · line 2575
Quoted 10 lines
fs/jbd2/journal.c +3
-1@@ -2559,11 +2568,14 @@ void jbd2_journal_put_journal_head(strucJ_ASSERT_JH (jh, jh->b_jcount >0 ); --jh->b_jcount;if (!jh->b_jcount) { -__journal_remove_journal_head (bh); +size_t b_size =__journal_remove_journal_head (bh); +jbd_unlock_bh_journal_head (bh); +journal_release_journal_head (jh, b_size);__brelse (bh);
The bh is pinned until you call __brelse(bh) above and bh->b_size doesn't change during the lifetime of the buffer. So there's no need of fetching bh->b_size in __journal_remove_journal_head() and passing it back. You can just:
journal_release_journal_head(jh, bh->b_size);
fs/jbd2/journal.c void jbd2_journal_put_journal_head(struc · line 2580
Quoted 6 lines
fs/jbd2/journal.c +2
-1- }else + }else {jbd_unlock_bh_journal_head (bh); + } }
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [patch V2 7/7] fs/jbd2: Free journal head outside of locked region
Quoted 22 lines · Jan Kara
Quoted 3 lines · Thomas Gleixner
On
PREEMPT_RTbit-spinlocks have the same semantics as on PREEMPT_RT=n, i.e. they disable preemption. That means functions which are not safe to be called in preempt disabled context on RT trigger amight_sleep()assert.Looks mostly good. Just a small suggestion for simplification below:
fs/jbd2/journal.c void jbd2_journal_put_journal_head(struc · line 2575
Quoted 10 lines
fs/jbd2/journal.c +3
-1@@ -2559,11 +2568,14 @@ void jbd2_journal_put_journal_head(strucJ_ASSERT_JH (jh, jh->b_jcount >0 ); --jh->b_jcount;if (!jh->b_jcount) { -__journal_remove_journal_head (bh); +size_t b_size =__journal_remove_journal_head (bh); +jbd_unlock_bh_journal_head (bh); +journal_release_journal_head (jh, b_size);__brelse (bh);The bh is pinned until you call
__brelse(bh)above andbh->b_sizedoesn't change during the lifetime of the buffer. So there's no need of fetchingbh->b_sizein__journal_remove_journal_head()and passing it back. You can just:
Quoted 1 line
journal_release_journal_head(jh, bh->b_size);
Ah. Nice. I assumed that this would be possible, but then my ignorance induced paranoia won.
Thanks,
tglx
Hi Thomas,
did you look into killing bіt spinlocks as a public API instead?
The main users seems to be buffer heads, which are so bloated that an extra spinlock doesn't really matter anyway.
The list_bl and rhashtable uses kinda make sense to be, but they are pretty nicely abstracted away anyway. The remaining users look pretty questionable to start with.
Christoph,
Quoted 1 line · Christoph Hellwig
did you look into killing bіt spinlocks as a public API instead?
Last time I did, there was resistance :)
But I'm happy to try again.
Quoted 6 lines
The main users seems to be buffer heads, which are so bloated that an extra spinlock doesn't really matter anyway.
The
list_bland rhashtable uses kinda make sense to be, but they are pretty nicely abstracted away anyway. The remaining users look pretty questionable to start with.
What about the page lock?
mm/slub.c: bit_spin_lock(PG_locked, &page->flags);
Thanks,
tglx
Quoted 1 line · Thomas Gleixner
Last time I did, there was resistance :)
Do you have a pointer? Note that in the buffer head case maybe a hash lock based on the page address is even better, as we only ever use the lock in the first buffer head of a page anyway..
Quoted 3 lines
What about the page lock?
mm/slub.c: bit_spin_lock(PG_locked, &page->flags);
One caller ouf of a gazillion that spins on the page lock instead of sleepign on it like everyone else. That should not have passed your smell test to start with :)
Quoted 6 lines · Christoph Hellwig
Quoted 1 line · Thomas Gleixner
Last time I did, there was resistance :)
Do you have a pointer? Note that in the buffer head case maybe a hash lock based on the page address is even better, as we only ever use the lock in the first buffer head of a page anyway..
I need to search my archives, but I'm on a spotty and slow connection right now. Will do so when back home.
Quoted 7 lines
Quoted 3 lines
What about the page lock?
mm/slub.c: bit_spin_lock(PG_locked, &page->flags);One caller ouf of a gazillion that spins on the page lock instead of sleepign on it like everyone else. That should not have passed your smell test to start with :)
I surely stared at it, but that cannot sleep. It's in the middle of a preempt and interrupt disabled region and used on architectures which do not support CMPXCHG_DOUBLE and ALIGNED_STRUCT_PAGE ...
Thanks,
tglx
Quoted 9 lines · Thomas Gleixner
Quoted 5 lines
Quoted 1 line
mm/slub.c: bit_spin_lock(PG_locked, &page->flags);One caller ouf of a gazillion that spins on the page lock instead of sleepign on it like everyone else. That should not have passed your smell test to start with :)
I surely stared at it, but that cannot sleep. It's in the middle of a preempt and interrupt disabled region and used on architectures which do not support
CMPXCHG_DOUBLEandALIGNED_STRUCT_PAGE...
I know. But the problem here is that normally PG_locked is used together with wait_on_page_bit_*, but this one instances uses the bit spinlock helpers. This is the equivalent of calling spin_lock on a struct mutex rather than having a mutex_lock_spin helper for this case. Does SLUB work on -rt at all?
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Thomas Gleixner
Quoted 5 lines
Quoted 1 line
mm/slub.c: bit_spin_lock(PG_locked, &page->flags);One caller ouf of a gazillion that spins on the page lock instead of sleepign on it like everyone else. That should not have passed your smell test to start with :)
Quoted 8 lines
Quoted 3 lines
I surely stared at it, but that cannot sleep. It's in the middle of a preempt and interrupt disabled region and used on architectures which do not support
CMPXCHG_DOUBLEandALIGNED_STRUCT_PAGE...I know. But the problem here is that normally
PG_lockedis used together with wait_on_page_bit_*, but this one instances uses the bit spinlock helpers. This is the equivalent of callingspin_lockon a struct mutex rather than having amutex_lock_spinhelper for this case.
Yes, I know :(
Quoted 1 line
Does SLUB work on -rt at all?
It's the only allocator we support with a few tweaks :)
Thanks,
tglx
Quoted 6 lines · Thomas Gleixner
Quoted 4 lines
I know. But the problem here is that normally
PG_lockedis used together with wait_on_page_bit_*, but this one instances uses the bit spinlock helpers. This is the equivalent of callingspin_lockon a struct mutex rather than having amutex_lock_spinhelper for this case.Yes, I know :(
But this means we should exclude slub from the bit_spin_lock removal. It really should use it's own version of it anyhow insted of pretending that the page lock is a bit spinlock.
Quoted 4 lines
Quoted 2 lines
Does SLUB work on -rt at all?
It's the only allocator we support with a few tweaks :)
What do you do about this particular piece of code there?
Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · Thomas Gleixner
Quoted 4 lines
I know. But the problem here is that normally
PG_lockedis used together with wait_on_page_bit_*, but this one instances uses the bit spinlock helpers. This is the equivalent of callingspin_lockon a struct mutex rather than having amutex_lock_spinhelper for this case.Yes, I know :(
But this means we should exclude slub from the
bit_spin_lockremoval. It really should use it's own version of it anyhow insted of pretending that the page lock is a bit spinlock.
But PG_locked isn't used as a mutex _when the page is allocated by slab_. Yes, every other user uses PG_locked as a mutex, but I don't see why that should constrain slub's usage of PG_locked.
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines
Quoted 1 line
Does SLUB work on -rt at all?
It's the only allocator we support with a few tweaks :)
What do you do about this particular piece of code there?
This part remains untouched. This "lock" is acquired within ->list_lock which is a raw_spinlock_t and disables preemption/interrupts on -RT.
Sebastian