mm: Speedup page cache truncation
Subsystems (13)
Jan Kara
maintainerv2 cover mm: Speedup page cache truncation
Hello,
this is a second version of my patches to avoid clearing marks from xas_store() and thus fix regression in page cache truncation.
Changes since v1
- rebased on 5.7-rc2
- drop xas_for_each_marked() fix as it was already merged
- reworked the whole series based on Matthew's feedback - we now create new function xas_store_noinit() and use it instead of changing xas_store() behavior. Note that for xas_store_range() and __xa_cmpxchg() I didn't bother to change names although they stop clearing marks as well. This is because there are only very few callers so it's easy to verify them, also chances of a clash with other patch introducing new callers are very small.
Original motivation:
Conversion of page cache to xarray (commit 69b6c1319b6 "mm: Convert truncate to XArray" in particular) has regressed performance of page cache truncation by about 10% (see my original report here [1]). This patch series aims at improving the truncation to get some of that regression back.
The first patch fixes a long standing bug with xas_for_each_marked() that I've uncovered when debugging my patches. The remaining patches then work towards the ability to stop clearing marks in xas_store() which improves truncation performance by about 6%.
The patches have passed radix_tree tests in tools/testing and also fstests runs for ext4 & xfs.
Honza
Jan Kara
maintainerPATCH v2 1/23 xarray: Remove stale comment
Since commit 7e934cf5ace1 "xarray: Fix early termination of xas_for_each_marked" the comment is no longer relevant since xas_for_each_marked() can cope with marked NULL entries. Remove the comment.
--- lib/xarray.c | 7 ------- 1 file changed, 7 deletions(-)
lib/xarray.c +0 -7
@@ -803,13 +803,6 @@ void *xas_store(struct xa_state *xas, void *entry)xas_init_marks (xas);for (;;) { -/* -* Must clear the marks before setting the entry to NULL, -* otherwise xas_for_each_marked may find a NULL entry and -* stop early. rcu_assign_pointer contains a release barrier -* so the mark clearing will appear to happen before the -* entry is set to NULL. -*/ rcu_assign_pointer (*slot, entry);if (xa_is_node (next) && (!node || node->shift))xas_free_nodes (xas,xa_to_node (next));
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 2/23 xarray: Provide xas_erase() and xas_store_noinit() helpers
In reply to Jan KaraCurrently xas_store() clears marks when stored value is NULL. This is somewhat counter-intuitive and also causes measurable performance impact when mark clearing is not needed (e.g. because marks are already clear). So provide xas_erase() helper (similarly to existing xa_erase()) which stores NULL at given index and also takes care of clearing marks. We also introduce xas_store_noinit() helper that works like xas_store() except that it does not initialize marks (and thus has better performance). In the following patches, all callers of xas_store() will be converted either to xas_erase() or xas_store_noinit().
--- include/linux/xarray.h | 2 ++ lib/xarray.c | 59 +++++++++++++++++++++++++++++++++++++------------- 2 files changed, 46 insertions(+), 15 deletions(-)
include/linux/xarray.h +2 -0
@@ -1491,6 +1491,8 @@ static inline bool xas_retry(struct xa_state *xas, const void *entry)void *xas_load (struct xa_state *);void *xas_store (struct xa_state *,void *entry); +void *xas_store_noinit (struct xa_state *,void *entry); +void *xas_erase (struct xa_state *);void *xas_find (struct xa_state *,unsigned long max);void *xas_find_conflict (struct xa_state *);
lib/xarray.c +44 -15
85 lines not on this page · raw message
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 3/23 xarray: Use xas_store_noinit() in __xa_store, __xa_insert, __xa_alloc
In reply to Jan KaraThese three functions never store NULL in the array so xas_store() is equivalent to xas_store_noinit(). Replace it.
--- lib/xarray.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
lib/xarray.c +5 -5
@@ -1079,7 +1079,7 @@ EXPORT_SYMBOL_GPL(__xas_next);* * If no entry is found and the array is smaller than @max, the iterator * is set to the smallest index not yet in the array. This allows @xas -* to be immediately passed to xas_store(). +* to be immediately passed to xas_store_noinit(). * * Return: The entry, if found, otherwise %NULL. */
@@ -1145,7 +1145,7 @@ EXPORT_SYMBOL_GPL(xas_find);* If no marked entry is found and the array is smaller than @max, @xas is * set to the bounds state and xas->xa_index is set to the smallest index * not yet in the array. This allows @xas to be immediately passed to -* xas_store(). +* xas_store_noinit(). * * If no entry is found before @max is reached, @xas is set to the restart * state.
@@ -1412,7 +1412,7 @@ void *__xa_store(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp) entry =XA_ZERO_ENTRY ;do { - curr =xas_store (&xas, entry); + curr =xas_store_noinit (&xas, entry);if (xa_track_free (xa))xas_clear_mark (&xas,XA_FREE_MARK ); }while (__xas_nomem (&xas, gfp));
@@ -1517,7 +1517,7 @@ int __xa_insert(struct xarray *xa, unsigned long index, void *entry, gfp_t gfp) do { curr = xas_load(&xas); if (!curr) { - xas_store(&xas, entry); + xas_store_noinit(&xas, entry); if (xa_track_free(xa)) xas_clear_mark(&xas, XA_FREE_MARK); } else {
@@ -1653,7 +1653,7 @@ int __xa_alloc(struct xarray *xa, u32 *id, void *entry,xas_set_err (&xas, -EBUSY ); else *id = xas.xa_index; -xas_store (&xas, entry); +xas_store_noinit (&xas, entry);xas_clear_mark (&xas,XA_FREE_MARK ); }while (__xas_nomem (&xas, gfp));
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 4/23 xarray: Switch xa_store_range() to use xas_store_noinit()
In reply to Jan KaraThere is only a single user of xa_store_range() and that is mm/memremap.c which does not use marks at all. Just switch xa_store_range() to use xas_store_noinit() to avoid implicit (and unnecessary for all users) initialization of xarray marks.
--- lib/xarray.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/xarray.c +1 -1
@@ -1602,7 +1602,7 @@ void *xa_store_range(struct xarray *xa, unsigned long first, } do { xas_set_range(&xas, first, last); - xas_store(&xas, entry); + xas_store_noinit(&xas, entry); if (xas_error(&xas)) goto unlock; first += xas_size(&xas);
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 5/23 xarray: Use xas_erase() in __xa_erase()
In reply to Jan KaraIn this case we want to erase element from the array. Use xas_erase() for it.
--- lib/xarray.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/xarray.c +1 -1
@@ -1358,7 +1358,7 @@ EXPORT_SYMBOL(xas_erase);void *__xa_erase (struct xarray *xa,unsigned long index) {XA_STATE (xas, xa, index); -return xas_result (&xas,xas_store (&xas,NULL )); +return xas_result (&xas,xas_erase (&xas)); }EXPORT_SYMBOL (__xa_erase);
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 6/23 xarray: Explicitely set XA_FREE_MARK in __xa_cmpxchg()
In reply to Jan Kara__xa_cmpxchg() relies on xas_store() to set XA_FREE_MARK when storing NULL into xarray that has free tracking enabled. Make the setting of XA_FREE_MARK explicit similarly as its clearing currently is.
--- lib/xarray.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
lib/xarray.c +6 -2
@@ -1479,8 +1479,12 @@ void *__xa_cmpxchg(struct xarray *xa, unsigned long index, curr =xas_load (&xas);if (curr == old) {xas_store (&xas, entry); -if (xa_track_free (xa) && entry && !curr) -xas_clear_mark (&xas,XA_FREE_MARK ); +if (xa_track_free (xa)) { +if (entry && !curr) +xas_clear_mark (&xas,XA_FREE_MARK ); +else if (!entry && curr) +xas_set_mark (&xas,XA_FREE_MARK ); + } } }while (__xas_nomem (&xas, gfp));
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 7/23 xarray: Switch __xa_cmpxchg() to use xas_store_noinit()
In reply to Jan KaraCurrently there are four places that end up calling into __xa_cmpxchg(). Two in drivers/infiniband/hw/mlx5/odp.c, one in fs/erofs/utils.c and one in mm/shmem.c. The xarray in the first three places do not contain any marks ever, the fourth place originally used radix_tree_delete_item() which didn't touch marks either. So we are safe to switch __xa_cmpxchg() to use xas_store_noinit(). Also document that __xa_cmpxchg() now does not touch marks at all - i.e., it has ordinary store semantics without specialcasing NULL value. If someone ever needs xa_cmpxchg() equivalent that would really cause mark clearing on storing NULL, we can create xa_erase_item() function.
--- lib/xarray.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
lib/xarray.c +3 -2
@@ -1460,7 +1460,8 @@ EXPORT_SYMBOL(xa_store);* * You must already be holding the xa_lock when calling this function. * It will drop the lock if needed to allocate memory, and then reacquire -* it afterwards. +* it afterwards. The call does not change any xarray marks except for +* XA_FREE_MARK if free tracking is enabled. * * Context: Any context. Expects xa_lock to be held on entry. May * release and reacquire xa_lock if @gfp flags permit.
@@ -1478,7 +1479,7 @@ void *__xa_cmpxchg(struct xarray *xa, unsigned long index, do { curr = xas_load(&xas); if (curr == old) { - xas_store(&xas, entry); + xas_store_noinit(&xas, entry); if (xa_track_free(xa)) { if (entry && !curr) xas_clear_mark(&xas, XA_FREE_MARK);
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 8/23 dax: Use xas_erase() in __dax_invalidate_entry()
In reply to Jan KaraWhen truncating DAX entry, we need to clear all the outstanding marks for the entry. Use dax_erase() instead of dax_store().
--- fs/dax.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/dax.c +1 -1
@@ -643,7 +643,7 @@ static int __dax_invalidate_entry(struct address_space *mapping, xas_get_mark(&xas, PAGECACHE_TAG_TOWRITE))) goto out;dax_disassociate_entry (entry, mapping, trunc); -xas_store (&xas,NULL ); +xas_erase (&xas); mapping->nrexceptional--; ret =1 ; out:
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 9/23 dax: Use dax_store_noinit() in grab_mapping_entry()
In reply to Jan KaraAlthough we store NULL here, there aren't any marks set on the entry so use xas_store_noinit() to save some time when trying to reinitialize marks. Furthermore we actually overwrite the entry shortly afterwards with new PTE entry anyway.
--- fs/dax.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
fs/dax.c +5 -1
@@ -523,7 +523,11 @@ static void *grab_mapping_entry(struct xa_state *xas, }dax_disassociate_entry (entry, mapping, false); -xas_store (xas,NULL );/* undo the PMD join */ +/* +* Remove PMD entry. There should be no marks set for the entry +* so don't bother reinitializing them with xas_erase(). +*/ +xas_store_noinit (xas,NULL );dax_wake_entry (xas, entry, true); mapping->nrexceptional--; entry =NULL ;
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 10/23 dax: Convert xas_store() to xas_store_noinit()
In reply to Jan KaraAll remaining users of xas_store() store non-NULL entries so xas_store() and xas_store_noinit() are equivalent. Replace xas_store() with xas_store_noinit().
--- fs/dax.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/dax.c +3 -3
@@ -283,7 +283,7 @@ static void dax_unlock_entry(struct xa_state *xas, void *entry)BUG_ON (dax_is_locked (entry));xas_reset (xas);xas_lock_irq (xas); - old =xas_store (xas, entry); + old =xas_store_noinit (xas, entry);xas_unlock_irq (xas);BUG_ON (!dax_is_locked (old));dax_wake_entry (xas, entry, false);
@@ -295,7 +295,7 @@ static void dax_unlock_entry(struct xa_state *xas, void *entry)static void *dax_lock_entry (struct xa_state *xas,void *entry) {unsigned long v =xa_to_value (entry); -return xas_store (xas,xa_mk_value (v |DAX_LOCKED )); +return xas_store_noinit (xas,xa_mk_value (v |DAX_LOCKED )); }static unsigned long dax_entry_size (void *entry)
@@ -923,7 +923,7 @@ static int dax_writeback_one(struct xa_state *xas, struct dax_device *dax_dev,*/ xas_reset (xas);xas_lock_irq (xas); -xas_store (xas, entry); +xas_store_noinit (xas, entry);xas_clear_mark (xas,PAGECACHE_TAG_DIRTY );dax_wake_entry (xas, entry, false);
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 11/23 mm: Use xas_erase() in page_cache_delete_batch()
In reply to Jan KaraWe need to clear marks when removing a page from xarray since there could be DIRTY or TOWRITE tags left for the page. Use xas_erase() to explicitely request mark clearing.
--- mm/filemap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/filemap.c +1 -1
@@ -333,7 +333,7 @@ static void page_cache_delete_batch(struct address_space *mapping,*/ if (page->index +compound_nr (page) -1 == xas.xa_index) i++; -xas_store (&xas,NULL ); +xas_erase (&xas); total_pages++; } mapping->nrpages -= total_pages;
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 12/23 mm: Use xas_erase() in collapse_file()
In reply to Jan KaraWhen undoing failed collapse of ordinary pages into a huge page, use xas_erase() to explicitly clear any xarray marks that may have been added to entries.
--- mm/khugepaged.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/khugepaged.c +1 -1
@@ -1819,7 +1819,7 @@ static void collapse_file(struct mm_struct *mm,break ; nr_none--;/* Put holes back where they were */ -xas_store (&xas,NULL ); +xas_erase (&xas);continue ; }
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 13/23 mm: Use xas_store_noinit() when storing non-NULL
In reply to Jan KaraWhen we store value different from NULL, xas_store_noinit() is equivalent to xas_store(). Transition these places to xas_store_noinit().
--- mm/filemap.c | 6 +++--- mm/khugepaged.c | 6 +++--- mm/migrate.c | 6 +++--- mm/shmem.c | 4 ++-- mm/swap_state.c | 2 +- 5 files changed, 12 insertions(+), 12 deletions(-)
mm/filemap.c +3 -3
@@ -134,7 +134,7 @@ static void page_cache_delete(struct address_space *mapping,VM_BUG_ON_PAGE (PageTail (page), page);VM_BUG_ON_PAGE (nr !=1 && shadow, page); -xas_store (&xas, shadow); +xas_store_noinit (&xas, shadow);xas_init_marks (&xas); page->mapping =NULL ;
@@ -803,7 +803,7 @@ int replace_page_cache_page(struct page *old, struct page *new, gfp_t gfp_mask) new->index = offset;xas_lock_irqsave (&xas, flags); -xas_store (&xas, new); +xas_store_noinit (&xas, new); old->mapping =NULL ;/* hugetlb pages do not participate in page cache accounting. */
@@ -856,7 +856,7 @@ static int __add_to_page_cache_locked(struct page *page, old =xas_load (&xas);if (old && !xa_is_value (old))xas_set_err (&xas, -EEXIST ); -xas_store (&xas, page); +xas_store_noinit (&xas, page);if (xas_error (&xas)) goto unlock;
mm/khugepaged.c +3 -3
@@ -1588,7 +1588,7 @@ static void collapse_file(struct mm_struct *mm, result =SCAN_FAIL ; goto xa_locked; } -xas_store (&xas, new_page); +xas_store_noinit (&xas, new_page); nr_none++;continue ; }
@@ -1724,7 +1724,7 @@ static void collapse_file(struct mm_struct *mm,list_add_tail (&page->lru, &pagelist);/* Finally, replace with the new page. */ -xas_store (&xas, new_page); +xas_store_noinit (&xas, new_page);continue ; out_unlock:unlock_page (page);
@@ -1828,7 +1828,7 @@ static void collapse_file(struct mm_struct *mm,/* Unfreeze the page. */ list_del (&page->lru);page_ref_unfreeze (page,2 ); -xas_store (&xas, page); +xas_store_noinit (&xas, page);xas_pause (&xas);xas_unlock_irq (&xas);unlock_page (page);
mm/migrate.c +3 -3
@@ -459,13 +459,13 @@ int migrate_page_move_mapping(struct address_space *mapping,SetPageDirty (newpage); } -xas_store (&xas, newpage); +xas_store_noinit (&xas, newpage);if (PageTransHuge (page)) {int i;for (i =1 ; i <HPAGE_PMD_NR ; i++) {xas_next (&xas); -xas_store (&xas, newpage); +xas_store_noinit (&xas, newpage); } }
@@ -536,7 +536,7 @@ int migrate_huge_page_move_mapping(struct address_space *mapping,get_page (newpage); -xas_store (&xas, newpage); +xas_store_noinit (&xas, newpage);page_ref_unfreeze (page, expected_count -1 );
mm/shmem.c +2 -2
@@ -361,7 +361,7 @@ static int shmem_replace_entry(struct address_space *mapping, item =xas_load (&xas);if (item != expected)return -ENOENT ; -xas_store (&xas, replacement); +xas_store_noinit (&xas, replacement);return 0 ; }
@@ -631,7 +631,7 @@ static int shmem_add_to_page_cache(struct page *page,if (xas_error (&xas)) goto unlock; next: -xas_store (&xas, page); +xas_store_noinit (&xas, page);if (++i < nr) {xas_next (&xas); goto next;
mm/swap_state.c +1 -1
@@ -133,7 +133,7 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, gfp_t gfp)for (i =0 ; i < nr; i++) {VM_BUG_ON_PAGE (xas.xa_index != idx + i, page);set_page_private (page + i, entry.val + i); -xas_store (&xas, page); +xas_store_noinit (&xas, page);xas_next (&xas); } address_space->nrpages += nr;
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 14/23 workingset: Use xas_store_noinit() to clear shadow entry
In reply to Jan KaraShadow entries have no marks. Use xas_store_noinit() to clear the entry so avoid unneeded initialization of the xarray marks. This provides a nice boost to truncate numbers. Sample benchmark showing time to truncate 128 files 1GB each on machine with 64GB of RAM (so about half of entries are shadow entries):
AVG STDDEV Vanilla 4.825s 0.036 Patched 4.516s 0.014
So we can see about 6% reduction in overall truncate time.
--- mm/truncate.c | 2 +- mm/workingset.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
mm/truncate.c +1 -1
@@ -39,7 +39,7 @@ static inline void __clear_shadow_entry(struct address_space *mapping,xas_set_update (&xas, workingset_update_node);if (xas_load (&xas) != entry)return ; -xas_store (&xas,NULL ); +xas_store_noinit (&xas,NULL ); mapping->nrexceptional--; }
mm/workingset.c +1 -1
@@ -537,7 +537,7 @@ static enum lru_status shadow_lru_isolate(struct list_head *item,* We could store a shadow entry here which was the minimum of the * shadow entries we were tracking ... */ -xas_store (&xas,NULL ); +xas_store_noinit (&xas,NULL );__inc_lruvec_slab_state (node,WORKINGSET_NODERECLAIM ); out_invalid:
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 15/23 swap: Use xas_erase() when removing page from swap cache
In reply to Jan KaraUse xas_erase() to explicitely clear xarray marks when removing swap cache pages from the i_mapping xarray.
--- mm/swap_state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/swap_state.c +1 -1
@@ -167,7 +167,7 @@ void __delete_from_swap_cache(struct page *page, swp_entry_t entry)VM_BUG_ON_PAGE (PageWriteback (page), page);for (i =0 ; i < nr; i++) { -void *entry =xas_store (&xas,NULL ); +void *entry =xas_erase (&xas);VM_BUG_ON_PAGE (entry != page, entry);set_page_private (page + i,0 );xas_next (&xas);
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 16/23 idr: Use xas_erase() in ida_destroy()
In reply to Jan KaraExplicitely clear marks (and set XA_MARK_FREE) in ida_destroy() by calling xas_erase() instead of relying on xas_store() on implicitely doing this.
--- lib/idr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/idr.c +1 -1
@@ -543,7 +543,7 @@ void ida_destroy(struct ida *ida)xas_for_each (&xas, bitmap,ULONG_MAX ) {if (!xa_is_value (bitmap))kfree (bitmap); -xas_store (&xas,NULL ); +xas_erase (&xas); }xas_unlock_irqrestore (&xas, flags); }
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 17/23 idr: Use xas_erase() in ida_free()
In reply to Jan KaraExplicitely clear marks in ida_free() by calling xas_erase() instead of relying on xas_store() on implicitely doing this.
--- lib/idr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/idr.c +1 -1
@@ -511,7 +511,7 @@ void ida_free(struct ida *ida, unsigned int id)if (bitmap_empty (bitmap->bitmap,IDA_BITMAP_BITS )) {kfree (bitmap); delete: -xas_store (&xas,NULL ); +xas_erase (&xas); } }xas_unlock_irqrestore (&xas, flags);
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 18/23 idr: Convert xas_store() to xas_store_noinit()
In reply to Jan KaraAll remaining users of xas_store() store non-NULL entries so xas_store() and xas_store_noinit() are equivalent. Replace xas_store() with xas_store_noinit().
--- lib/idr.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
lib/idr.c +4 -4
@@ -408,7 +408,7 @@ int ida_alloc_range(struct ida *ida, unsigned int min, unsigned int max, goto nospc;if (bit <BITS_PER_XA_VALUE ) { tmp |=1UL << bit; -xas_store (&xas,xa_mk_value (tmp)); +xas_store_noinit (&xas,xa_mk_value (tmp)); goto out; } }
@@ -418,7 +418,7 @@ int ida_alloc_range(struct ida *ida, unsigned int min, unsigned int max,if (!bitmap) goto alloc; bitmap->bitmap[0 ] = tmp; -xas_store (&xas, bitmap); +xas_store_noinit (&xas, bitmap);if (xas_error (&xas)) { bitmap->bitmap[0 ] =0 ; goto out;
@@ -446,7 +446,7 @@ int ida_alloc_range(struct ida *ida, unsigned int min, unsigned int max, goto alloc;__set_bit (bit, bitmap->bitmap); } -xas_store (&xas, bitmap); +xas_store_noinit (&xas, bitmap); } out:xas_unlock_irqrestore (&xas, flags);
@@ -502,7 +502,7 @@ void ida_free(struct ida *ida, unsigned int id) v &= ~(1UL << bit);if (!v) goto delete; -xas_store (&xas,xa_mk_value (v)); +xas_store_noinit (&xas,xa_mk_value (v)); }else {if (!test_bit (bit, bitmap->bitmap)) goto err;
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 19/23 testing: Use xas_erase() to remove entries from xarray
In reply to Jan KaraUse xas_erase() instead of xas_store(... NULL) for removing entries from xarray.
--- lib/test_xarray.c | 10 +++++----- tools/testing/radix-tree/test.c | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-)
lib/test_xarray.c +5 -5
@@ -318,7 +318,7 @@ static noinline void check_xa_shrink(struct xarray *xa)XA_BUG_ON (xa,xas_load (&xas) !=xa_mk_value (1 )); node = xas.xa_node;XA_BUG_ON (xa,xa_entry_locked (xa, node,0 ) !=xa_mk_value (0 )); -XA_BUG_ON (xa,xas_store (&xas,NULL ) !=xa_mk_value (1 )); +XA_BUG_ON (xa,xas_erase (&xas) !=xa_mk_value (1 ));XA_BUG_ON (xa,xa_load (xa,1 ) !=NULL );XA_BUG_ON (xa, xas.xa_node !=XAS_BOUNDS );XA_BUG_ON (xa,xa_entry_locked (xa, node,0 ) !=XA_RETRY_ENTRY );
@@ -488,13 +488,13 @@ static noinline void check_xas_erase(struct xarray *xa) }while (xas_nomem (&xas,GFP_KERNEL ));xas_lock (&xas); -xas_store (&xas,NULL ); +xas_erase (&xas);xas_set (&xas,0 ); j = i;xas_for_each (&xas, entry,ULONG_MAX ) {XA_BUG_ON (xa, entry !=xa_mk_index (j)); -xas_store (&xas,NULL ); +xas_erase (&xas); j++; }xas_unlock (&xas);
@@ -537,7 +537,7 @@ static noinline void check_multi_store_2(struct xarray *xa, unsigned long index,xas_lock (&xas);XA_BUG_ON (xa,xas_store (&xas,xa_mk_value (1 )) !=xa_mk_value (0 ));XA_BUG_ON (xa, xas.xa_index != index); -XA_BUG_ON (xa,xas_store (&xas,NULL ) !=xa_mk_value (1 )); +XA_BUG_ON (xa,xas_erase (&xas) !=xa_mk_value (1 ));xas_unlock (&xas);XA_BUG_ON (xa, !xa_empty (xa)); }
@@ -1582,7 +1582,7 @@ static noinline void shadow_remove(struct xarray *xa) xas.xa_offset = node->offset; xas.xa_shift = node->shift +XA_CHUNK_SHIFT ;xas_set_update (&xas, test_update_node); -xas_store (&xas,NULL ); +xas_erase (&xas); }xa_unlock (xa); }
tools/testing/radix-tree/test.c +1 -1
@@ -261,7 +261,7 @@ void item_kill_tree(struct xarray *xa)if (!xa_is_value (entry)) {item_free (entry, xas.xa_index); } -xas_store (&xas,NULL ); +xas_erase (&xas); }assert (xa_empty (xa));
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 20/23 testing: Use xas_store_noinit() for non-NULL entries
In reply to Jan KaraWhen we store value different from NULL, xas_store_noinit() is equivalent to xas_store(). Transition these places to xas_store_noinit().
--- lib/test_xarray.c | 28 ++++++++++++++-------------- tools/testing/radix-tree/iteration_check.c | 2 +- tools/testing/radix-tree/multiorder.c | 2 +- 3 files changed, 16 insertions(+), 16 deletions(-)
lib/test_xarray.c +14 -14
@@ -130,13 +130,13 @@ static noinline void check_xas_retry(struct xarray *xa)/* Make sure we can iterate through retry entries */ xas_lock (&xas);xas_set (&xas,0 ); -xas_store (&xas,XA_RETRY_ENTRY ); +xas_store_noinit (&xas,XA_RETRY_ENTRY );xas_set (&xas,1 ); -xas_store (&xas,XA_RETRY_ENTRY ); +xas_store_noinit (&xas,XA_RETRY_ENTRY );xas_set (&xas,0 );xas_for_each (&xas, entry,ULONG_MAX ) { -xas_store (&xas,xa_mk_index (xas.xa_index)); +xas_store_noinit (&xas,xa_mk_index (xas.xa_index)); }xas_unlock (&xas);
@@ -475,7 +475,7 @@ static noinline void check_xas_erase(struct xarray *xa)xas_set (&xas, j);do {xas_lock (&xas); -xas_store (&xas,xa_mk_index (j)); +xas_store_noinit (&xas,xa_mk_index (j));xas_unlock (&xas); }while (xas_nomem (&xas,GFP_KERNEL )); }
@@ -483,7 +483,7 @@ static noinline void check_xas_erase(struct xarray *xa)xas_set (&xas,ULONG_MAX );do {xas_lock (&xas); -xas_store (&xas,xa_mk_value (0 )); +xas_store_noinit (&xas,xa_mk_value (0 ));xas_unlock (&xas); }while (xas_nomem (&xas,GFP_KERNEL ));
@@ -517,7 +517,7 @@ static noinline void check_multi_store_1(struct xarray *xa, unsigned long index,XA_BUG_ON (xa,xa_load (xa, min -1 ) !=NULL );xas_lock (&xas); -XA_BUG_ON (xa,xas_store (&xas,xa_mk_index (min)) !=xa_mk_index (index)); +XA_BUG_ON (xa,xas_store_noinit (&xas,xa_mk_index (min)) !=xa_mk_index (index));xas_unlock (&xas);XA_BUG_ON (xa,xa_load (xa, min) !=xa_mk_index (min));XA_BUG_ON (xa,xa_load (xa, max -1 ) !=xa_mk_index (min));
@@ -535,7 +535,7 @@ static noinline void check_multi_store_2(struct xarray *xa, unsigned long index,xa_store_order (xa, index, order,xa_mk_value (0 ),GFP_KERNEL );xas_lock (&xas); -XA_BUG_ON (xa,xas_store (&xas,xa_mk_value (1 )) !=xa_mk_value (0 )); +XA_BUG_ON (xa,xas_store_noinit (&xas,xa_mk_value (1 )) !=xa_mk_value (0 ));XA_BUG_ON (xa, xas.xa_index != index);XA_BUG_ON (xa,xas_erase (&xas) !=xa_mk_value (1 ));xas_unlock (&xas);
@@ -854,7 +854,7 @@ static noinline void __check_store_iter(struct xarray *xa, unsigned long start,XA_BUG_ON (xa, entry >xa_mk_index (start + (1UL << order) -1 )); count++; } -xas_store (&xas,xa_mk_index (start)); +xas_store_noinit (&xas,xa_mk_index (start));xas_unlock (&xas);if (xas_nomem (&xas,GFP_KERNEL )) { count =0 ;
@@ -1365,7 +1365,7 @@ static noinline void xa_store_many_order(struct xarray *xa,if (xas_error (&xas)) goto unlock;for (i =0 ; i < (1U << order); i++) { -XA_BUG_ON (xa,xas_store (&xas,xa_mk_index (index + i))); +XA_BUG_ON (xa,xas_store_noinit (&xas,xa_mk_index (index + i)));xas_next (&xas); } unlock:
@@ -1420,7 +1420,7 @@ static noinline void check_create_range_4(struct xarray *xa,if (xas_error (&xas)) goto unlock;for (i =0 ; i < (1UL << order); i++) { -void *old =xas_store (&xas,xa_mk_index (base + i)); +void *old =xas_store_noinit (&xas,xa_mk_index (base + i));if (xas.xa_index == index)XA_BUG_ON (xa, old !=xa_mk_index (base + i));else
@@ -1594,9 +1594,9 @@ static noinline void check_workingset(struct xarray *xa, unsigned long index)do {xas_lock (&xas); -xas_store (&xas,xa_mk_value (0 )); +xas_store_noinit (&xas,xa_mk_value (0 ));xas_next (&xas); -xas_store (&xas,xa_mk_value (1 )); +xas_store_noinit (&xas,xa_mk_value (1 ));xas_unlock (&xas); }while (xas_nomem (&xas,GFP_KERNEL ));
@@ -1604,10 +1604,10 @@ static noinline void check_workingset(struct xarray *xa, unsigned long index)xas_lock (&xas);xas_next (&xas); -xas_store (&xas, &xas); +xas_store_noinit (&xas, &xas);XA_BUG_ON (xa, !list_empty (&shadow_nodes)); -xas_store (&xas,xa_mk_value (2 )); +xas_store_noinit (&xas,xa_mk_value (2 ));xas_unlock (&xas);XA_BUG_ON (xa,list_empty (&shadow_nodes));
tools/testing/radix-tree/iteration_check.c +1 -1
@@ -31,7 +31,7 @@ void my_item_insert(struct xarray *xa, unsigned long index) item->order = order;if (xas_find_conflict (&xas))continue ; -xas_store (&xas, item); +xas_store_noinit (&xas, item);xas_set_mark (&xas,TAG );break ; }
tools/testing/radix-tree/multiorder.c +1 -1
@@ -20,7 +20,7 @@ static int item_insert_order(struct xarray *xa, unsigned long index,do {xas_lock (&xas); -xas_store (&xas, item); +xas_store_noinit (&xas, item);xas_unlock (&xas); }while (xas_nomem (&xas,GFP_KERNEL ));
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 21/23 testing: Introduce xa_erase_order() and use it
In reply to Jan KaraIntroduce helper xa_erase_order() and call it from places that want to erase entries from xarray instead of using xa_store_order(). This also explicitely takes care of clearing possibly existing xarray marks (although no current test seems to need it).
--- lib/test_xarray.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
lib/test_xarray.c +12 -2
@@ -83,6 +83,16 @@ static void *xa_store_order(struct xarray *xa, unsigned long index,return curr; } +static void xa_erase_order (struct xarray *xa,unsigned long index, +unsigned order) +{ +XA_STATE_ORDER (xas, xa, index, order); + +xas_lock (&xas); +xas_erase (&xas); +xas_unlock (&xas); +} + static noinline void check_xa_err(struct xarray *xa) {XA_BUG_ON (xa,xa_err (xa_store_index (xa,0 ,GFP_NOWAIT )) !=0 );
@@ -608,13 +618,13 @@ static noinline void check_multi_store(struct xarray *xa)rcu_read_unlock ();/* We can erase multiple values with a single store */ -xa_store_order (xa,0 ,BITS_PER_LONG -1 ,NULL ,GFP_KERNEL ); +xa_erase_order (xa,0 ,BITS_PER_LONG -1 );XA_BUG_ON (xa, !xa_empty (xa));/* Even when the first slot is empty but the others aren't */ xa_store_index (xa,1 ,GFP_KERNEL );xa_store_index (xa,2 ,GFP_KERNEL ); -xa_store_order (xa,0 ,2 ,NULL ,GFP_KERNEL ); +xa_erase_order (xa,0 ,2 );XA_BUG_ON (xa, !xa_empty (xa));for (i =0 ; i < max_order; i++) {
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 22/23 testing: Switch xa_store_order() to xas_store_noinit()
In reply to Jan Karaxa_store_order() is currently only used for non-NULL entries. So xas_store_noinit() is equivalent to xas_store() and we can safely use it in xa_store_order.
--- lib/test_xarray.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/test_xarray.c +1 -1
@@ -76,7 +76,7 @@ static void *xa_store_order(struct xarray *xa, unsigned long index,do {xas_lock (&xas); - curr =xas_store (&xas, entry); + curr =xas_store_noinit (&xas, entry);xas_unlock (&xas); }while (xas_nomem (&xas, gfp));
Signature · 1 line
2.16.4
Jan Kara
maintainerPATCH v2 23/23 xarray: Remove xas_store()
In reply to Jan Karaxas_store() now has no users as every call site has been transitioned either to xas_erase() or to xas_store_noinit(). Remove xas_store() and all remaining traces of it.
--- Documentation/core-api/xarray.rst | 4 ++-- include/linux/xarray.h | 1 - lib/xarray.c | 14 ++++---------- 3 files changed, 6 insertions(+), 13 deletions(-)
Documentation/core-api/xarray.rst +2 -2
@@ -394,7 +394,7 @@ You can use xas_init_marks() to reset the marks on an entry to their default state. This is usually all marks clear, unless the XArray is marked with ``XA_FLAGS_TRACK_FREE``, in which case mark 0 is set and all other marks are clear. Replacing one entry with another using -xas_store() will not reset the marks on that entry; if you want +xas_store_noinit() will not reset the marks on that entry; if you want the marks reset, you should do that explicitly. The xas_load() will walk the xa_state as close to the entry
@@ -458,7 +458,7 @@ save substantial quantities of memory; for example tying 512 entries together will save over 4kB. You can create a multi-index entry by using XA_STATE_ORDER() -or xas_set_order() followed by a call to xas_store(). +or xas_set_order() followed by a call to xas_store_noinit(). Calling xas_load() with a multi-index xa_state will walk the xa_state to the right location in the tree, but the return value is not meaningful, potentially being an internal entry or ``NULL`` even when there
include/linux/xarray.h +0 -1
@@ -1490,7 +1490,6 @@ static inline bool xas_retry(struct xa_state *xas, const void *entry) }void *xas_load (struct xa_state *); -void *xas_store (struct xa_state *,void *entry);void *xas_store_noinit (struct xa_state *,void *entry);void *xas_erase (struct xa_state *);void *xas_find (struct xa_state *,unsigned long max);
lib/xarray.c +4 -10
@@ -625,7 +625,7 @@ static int xas_expand(struct xa_state *xas, void *head)* @allow_root: %true if we can store the entry in the root directly * * Most users will not need to call this function directly, as it is called -* by xas_store(). It is useful for doing conditional store operations +* by xas_store_noinit(). It is useful for doing conditional store operations * (see the xa_cmpxchg() implementation for an example). * * Return: If the slot already existed, returns the contents of this slot.
@@ -820,7 +820,7 @@ static void *__xas_store(struct xa_state *xas, void *entry, bool init_marks) }/** -* xas_store() - Store this entry in the XArray. +* xas_store_noinit() - Store this entry in the XArray. * @xas: XArray operation state. * @entry: New entry. *
@@ -828,17 +828,11 @@ static void *__xas_store(struct xa_state *xas, void *entry, bool init_marks)* function is essentially meaningless (it may be an internal entry or it * may be %NULL, even if there are non-NULL entries at some of the indices * covered by the range). This is not a problem for any current users, -* and can be changed if needed. +* and can be changed if needed. Note that this function does not affect +* xarray marks. * * Return: The old entry at this index. */ -void *xas_store (struct xa_state *xas,void *entry) -{ -return __xas_store (xas, entry, true); -} -EXPORT_SYMBOL_GPL (xas_store); - -/* This is like xas_store() but does not initialize marks on stored entry */ void *xas_store_noinit (struct xa_state *xas,void *entry) {return __xas_store (xas, entry, false);
Signature · 1 line
2.16.4