famfs: Fabric-Attached Memory File System
Subsystems (8)
John Groves
maintainerPATCH v8 1/1 famfs: Fabric-Attached Memory File System
This is a coordinated patch submission for famfs (Fabric-Attached Memory File System) across three repositories:
1. Linux kernel dax (cover + 7 patches) - dax fsdev driver + fuse/famfs integration
2. Linux kernel fs/fuse - famfs support into fuse. Depends on the dax patches
This bundle does not contain the user space libfuse and ndctl patches; those will be sent separately to the respective projects.
Note: in this version I split the kernel dax patches into a separate series from the kernel fuse patches. This is because those go through separate maintainers. The plan is for the dax patches to go into Ira's dax tree, and for Miklos to pull in that branch and apply the fuse series on top if it.
Each series is posted as a reply to this cover message, with individual patches replying to their respective series cover.
Overview
--------
Famfs exposes shared memory as a file system. It consumes shared memory from dax devices and provides memory-mappable files that map directly to the memory with no page cache involvement. Famfs differs from conventional file systems in fs-dax mode in that it handles in-memory metadata in a sharable way (which begins with never caching dirty shared metadata).
Famfs started as a standalone file system [1,2], but the consensus at LSFMM 2024 and 2025 [3,4] was that it should be ported into fuse.
The key performance requirement is that famfs must resolve mapping faults without upcalls. This is achieved by fully caching the file-to-devdax metadata for all active files via two fuse client/server message/response
pairs: GET_FMAP and GET_DAXDEV.
Patch Series Summary
--------------------
Linux Kernel dax (V8, xx patches): New fsdev driver (drivers/dax/fsdev.c) providing a devdax mode compatible with fs-dax. Devices can be switched among 'devdax', 'fsdev' and 'system-ram' modes via daxctl or sysfs.
Linux kernel fuse (V8, xx patches: Famfs integration adding GET_FMAP and GET_DAXDEV messages for caching file-to-dax mappings in the kernel.
Testing
-------
The famfs user space [5] includes comprehensive smoke and unit tests that exercise all three components together. The ndctl series includes a dedicated test for famfs mode transitions.
References
----------
[1] https://lore.kernel.org/linux-cxl/[email protected]/
[2] https://lore.kernel.org/linux-cxl/[email protected]/
[3] https://lwn.net/Articles/983105/ (LSFMM 2024)
[4] https://lwn.net/Articles/1020170/ (LSFMM 2025)
[5] https://famfs.org (famfs user space)
[6] https://lore.kernel.org/linux-cxl/[email protected]/ (V2)
[7] https://lore.kernel.org/linux-fsdevel/[email protected]/T/#m0000d8c00290f48c086b8b176c7525e410f8508c (related ndctl series) --
John Groves
John Groves
maintainerv8 cover dax: prepare for famfs
This patch series along with the bundled patches to fuse are available as a git tag at [0].
Changes v7 -> v8
- dax: Added a devm action to clear folio state when unbinding fsdev.c (thanks Allison)
- Added a missing device_lock() in fs_dax_get() (thanks Dave)
- Re-factored some __free blocks for inline declaration
- Used FIELD_PREP where appropriate
- Minor doc edits
Description:
This patch series introduces the required dax support for famfs. Previous versions of the famfs series included both dax and fuse patches. This series separates them into separate patch series' (and the fuse series dependends on this dax series).
The famfs user space code can be found at [1]
Dax Overview:
This series introduces a new "famfs mode" of devdax, whose driver is drivers/dax/fsdev.c. This driver supports dax_iomap_rw() and dax_iomap_fault() calls against a character dax instance. A dax device now can be converted among three modes: 'system-ram', 'devdax' and 'famfs' via daxctl or sysfs (e.g. unbind devdax and bind famfs instead).
In famfs mode, a dax device initializes its pages consistent with the fsdaxmode of pmem. Raw read/write/mmap are not supported in this mode, but famfs is happy in this mode - using dax_iomap_rw() for read/write and dax_iomap_fault() for mmap faults.
Changes v6 -> v7
- Fixed a regression in famfs_interleave_fileofs_to_daxofs() that was reported by Intel's kernel test robot
- Added a check in __fsdev_dax_direct_access() for negative return from pgoff_to_phys(), which would indicate an out-of-range offset
- Fixed a bug in __famfs_meta_free(), where not all interleaved extents were freed
- Added chunksize alignment checks in famfs_fuse_meta_alloc() and famfs_interleave_fileofs_to_daxofs() as interleaved chunks must be PTE or PMD aligned
- Simplified famfs_file_init_dax() a bit
- Re-ran CM's kernel code review prompts on the entire series and fixed several minor issues
Changes v4 -> v5 -> v6
- None. Re-sending due to technical difficulties
Changes v3 [9] -> v4
- The patch "dax: prevent driver unbind while filesystem holds device" has been dropped. Dan Williams indicated that the favored behavior is for a file system to stop working if an underlying driver is unbound, rather than preventing the unbind.
- The patch "famfs_fuse: Famfs mount opt: -o shadow=<shadowpath>" has been dropped. Found a way for the famfs user space to do without the -o opt (via getxattr).
- Squashed the fs/fuse/Kconfig patch into the first subsequent patch that needed the change
("famfs_fuse: Basic fuse kernel ABI enablement for famfs")
- Many review comments addressed.
- Addressed minor kerneldoc infractions reported by test robot.
Changes v2 [7] -> v3
- Dax: Completely new fsdev driver (drivers/dax/fsdev.c) replaces the dev_dax_iomap modifications to bus.c/device.c. Devdax devices can now be switched among 'devdax', 'famfs' and 'system-ram' modes via daxctl or sysfs.
- Dax: fsdev uses MEMORY_DEVICE_FS_DAX type and leaves folios at order-0 (no vmemmap_shift), allowing fs-dax to manage folio lifecycles dynamically like pmem does.
- Dax: The "poisoned page" problem is properly fixed via fsdev_clear_folio_state(), which clears stale mapping/compound state when fsdev binds. The temporary WARN_ON_ONCE workaround in fs/dax.c has been removed.
- Dax: Added dax_set_ops() so fsdev can set dax_operations at bind time (and clear them on unbind), since the dax_device is created before we know which driver will bind.
- Dax: Added custom bind/unbind sysfs handlers; unbind return -EBUSY if a filesystem holds the device, preventing unbind while famfs is mounted.
- Fuse: Famfs mounts now require that the fuse server/daemon has CAP_SYS_RAWIO because they expose raw memory devices.
- Fuse: Added DAX address_space_operations with noop_dirty_folio since famfs is memory-backed with no writeback required.
- Rebased to latest kernels, fully compatible with Alistair Popple et. al's recent dax refactoring.
- Ran this series through Chris Mason's code review AI prompts to check for issues - several subtle problems found and fixed.
- Dropped RFC status - this version is intended to be mergeable.
Changes v1 [8] -> v2:
- The GET_FMAP message/response has been moved from LOOKUP to OPEN, as was the pretty much unanimous consensus.
- Made the response payload to GET_FMAP variable sized (patch 12)
- Dodgy kerneldoc comments cleaned up or removed.
- Fixed memory leak of fc->shadow in patch 11 (thanks Joanne)
- Dropped many pr_debug and pr_notice calls
References
[0] - https://github.com/jagalactic/linux/tree/famfs-v8 (this patch set)
[1] - https://famfs.org (famfs user space)
[2] - https://lore.kernel.org/linux-cxl/[email protected]/
[3] - https://lore.kernel.org/linux-cxl/[email protected]/
[4] - https://lwn.net/Articles/983105/ (lsfmm 2024)
[5] - https://lwn.net/Articles/1020170/ (lsfmm 2025)
[6] - https://lore.kernel.org/linux-cxl/cover.8068ad144a7eea4a813670301f4d2a86a8e68ec4.1740713401.git-series.apopple@nvidia.com/
[7] - https://lore.kernel.org/linux-fsdevel/[email protected]/ (famfs fuse v2)
[8] - https://lore.kernel.org/linux-fsdevel/[email protected]/ (famfs fuse v1)
[9] - https://lore.kernel.org/linux-fsdevel/[email protected]/T/#mb2c868801be16eca82dab239a1d201628534aea7 (famfs fuse v3)
John Groves (8):
dax: move dax_pgoff_to_phys from [drivers/dax/] device.c to bus.c
dax: Factor out dax_folio_reset_order() helper
dax: add fsdev.c driver for fs-dax on character dax
dax: Save the kva from memremap
dax: Add dax_operations for use by fs-dax on fsdev dax
dax: Add dax_set_ops() for setting dax_operations at bind time
dax: Add fs_dax_get() func to prepare dax for fs-dax usage
dax: export dax_dev_get()
MAINTAINERS | 8 + drivers/dax/Makefile | 6 + drivers/dax/bus.c | 30 +++- drivers/dax/bus.h | 3 + drivers/dax/dax-private.h | 3 + drivers/dax/device.c | 23 --- drivers/dax/fsdev.c | 353 ++++++++++++++++++++++++++++++++++++++ drivers/dax/super.c | 107 +++++++++++- fs/dax.c | 61 +++++-- include/linux/dax.h | 21 ++- 10 files changed, 565 insertions(+), 50 deletions(-) create mode 100644 drivers/dax/fsdev.c
base-commit: f338e77383789c0cae23ca3d48adcc5e9e137e3c
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 1/8 dax: move dax_pgoff_to_phys from [drivers/dax/] device.c to bus.c
This function will be used by both device.c and fsdev.c, but both are loadable modules. Moving to bus.c puts it in core and makes it available to both.
No code changes - just relocated.
--- drivers/dax/bus.c | 24 ++++++++++++++++++++++++ drivers/dax/device.c | 23 ----------------------- 2 files changed, 24 insertions(+), 23 deletions(-)
drivers/dax/bus.c +24 -0
@@ -1417,6 +1417,30 @@ static const struct device_type dev_dax_type = { .groups = dax_attribute_groups, }; +/* see "strong" declaration in tools/testing/nvdimm/dax-dev.c */ +__weak phys_addr_t dax_pgoff_to_phys (struct dev_dax *dev_dax,pgoff_t pgoff, +unsigned long size) +{ +int i; + +for (i =0 ; i < dev_dax->nr_range; i++) { +struct dev_dax_range *dax_range = &dev_dax->ranges[i]; +struct range *range = &dax_range->range; +unsigned long long pgoff_end; +phys_addr_t phys; + + pgoff_end = dax_range->pgoff +PHYS_PFN (range_len (range)) -1 ; +if (pgoff < dax_range->pgoff || pgoff > pgoff_end) +continue ; + phys =PFN_PHYS (pgoff - dax_range->pgoff) + range->start; +if (phys + size -1 <= range->end) +return phys; +break ; + } +return -1 ; +} +EXPORT_SYMBOL_GPL (dax_pgoff_to_phys); +static struct dev_dax *__devm_create_dev_dax (struct dev_dax_data *data) {struct dax_region *dax_region = data->dax_region;
drivers/dax/device.c +0 -23
@@ -57,29 +57,6 @@ static int check_vma(struct dev_dax *dev_dax, struct vm_area_struct *vma, vma->vm_file, func); } -/* see "strong" declaration in tools/testing/nvdimm/dax-dev.c */ -__weak phys_addr_t dax_pgoff_to_phys (struct dev_dax *dev_dax,pgoff_t pgoff, -unsigned long size) -{ -int i; - -for (i =0 ; i < dev_dax->nr_range; i++) { -struct dev_dax_range *dax_range = &dev_dax->ranges[i]; -struct range *range = &dax_range->range; -unsigned long long pgoff_end; -phys_addr_t phys; - - pgoff_end = dax_range->pgoff +PHYS_PFN (range_len (range)) -1 ; -if (pgoff < dax_range->pgoff || pgoff > pgoff_end) -continue ; - phys =PFN_PHYS (pgoff - dax_range->pgoff) + range->start; -if (phys + size -1 <= range->end) -return phys; -break ; - } -return -1 ; -} -static void dax_set_mapping (struct vm_fault *vmf,unsigned long pfn,unsigned long fault_size) {
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 2/8 dax: Factor out dax_folio_reset_order() helper
In reply to John GrovesFrom: John Groves <[email protected]>
Both fs/dax.c:dax_folio_put() and drivers/dax/fsdev.c:fsdev_clear_folio_state() (the latter coming in the next commit after this one) contain nearly identical code to reset a compound DAX folio back to order-0 pages. Factor this out into a shared helper function.
The new dax_folio_reset_order() function:
- Clears the folio's mapping and share count
- Resets compound folio state via folio_reset_order()
- Clears PageHead and compound_head for each sub-page
- Restores the pgmap pointer for each resulting order-0 folio
- Returns the original folio order (for callers that need to advance by that many pages)
This simplifies fsdev_clear_folio_state() from ~50 lines to ~15 lines while maintaining the same functionality in both call sites.
--- fs/dax.c | 60 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 42 insertions(+), 18 deletions(-)
fs/dax.c +42 -18
@@ -378,6 +378,45 @@ static void dax_folio_make_shared(struct folio *folio) folio->share =1 ; } +/** +* dax_folio_reset_order - Reset a compound DAX folio to order-0 pages +* @folio: The folio to reset +* +* Splits a compound folio back into individual order-0 pages, +* clearing compound state and restoring pgmap pointers. +* +* Returns: the original folio order (0 if already order-0) +*/ +int dax_folio_reset_order (struct folio *folio) +{ +struct dev_pagemap *pgmap =page_pgmap (&folio->page); +int order =folio_order (folio); +int i; + + folio->mapping =NULL ; + folio->share =0 ; + +if (!order) { + folio->pgmap = pgmap; +return 0 ; + } + +folio_reset_order (folio); + +for (i =0 ; i < (1UL << order); i++) { +struct page *page =folio_page (folio, i); +struct folio *f = (struct folio *)page; + +ClearPageHead (page); +clear_compound_head (page); + f->mapping =NULL ; + f->share =0 ; + f->pgmap = pgmap; + } + +return order; +} +static inline unsigned long dax_folio_put (struct folio *folio) {unsigned long ref;
@@ -391,28 +430,13 @@ static inline unsigned long dax_folio_put(struct folio *folio)if (ref)return ref; - folio->mapping =NULL ; - order =folio_order (folio); -if (!order) -return 0 ; -folio_reset_order (folio); + order =dax_folio_reset_order (folio); +/* Debug check: verify refcounts are zero for all sub-folios */ for (i =0 ; i < (1UL << order); i++) { -struct dev_pagemap *pgmap =page_pgmap (&folio->page);struct page *page =folio_page (folio, i); -struct folio *new_folio = (struct folio *)page; -ClearPageHead (page); -clear_compound_head (page); - - new_folio->mapping =NULL ; -/* -* Reset pgmap which was over-written by -* prep_compound_page(). -*/ - new_folio->pgmap = pgmap; - new_folio->share =0 ; -WARN_ON_ONCE (folio_ref_count (new_folio)); +WARN_ON_ONCE (folio_ref_count ((struct folio *)page)); }return ref;
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 3/8 dax: add fsdev.c driver for fs-dax on character dax
In reply to John GrovesThe new fsdev driver provides pages/folios initialized compatibly with fsdax - normal rather than devdax-style refcounting, and starting out with order-0 folios.
When fsdev binds to a daxdev, it is usually (always?) switching from the devdax mode (device.c), which pre-initializes compound folios according to its alignment. Fsdev uses fsdev_clear_folio_state() to switch the folios into a fsdax-compatible state.
A side effect of this is that raw mmap doesn't (can't?) work on an fsdev dax instance. Accordingly, The fsdev driver does not provide raw mmap - devices must be put in 'devdax' mode (drivers/dax/device.c) to get raw mmap capability.
In this commit is just the framework, which remaps pages/folios compatibly with fsdax.
Enabling dax changes:
- bus.h: add DAXDRV_FSDEV_TYPE driver type
- bus.c: allow DAXDRV_FSDEV_TYPE drivers to bind to daxdevs
- dax.h: prototype inode_dax(), which fsdev needs
--- MAINTAINERS | 8 ++ drivers/dax/Makefile | 6 + drivers/dax/bus.c | 4 + drivers/dax/bus.h | 1 + drivers/dax/fsdev.c | 253 +++++++++++++++++++++++++++++++++++++++++++ fs/dax.c | 1 + include/linux/dax.h | 3 + 7 files changed, 276 insertions(+) create mode 100644 drivers/dax/fsdev.c
MAINTAINERS +8 -0
@@ -7298,6 +7298,14 @@ L: [email protected] S: Supported F: drivers/dax/ +DEVICE DIRECT ACCESS (DAX) [fsdev_dax] +M: John Groves <[email protected]> +M: John Groves <[email protected]> +L: [email protected] +L: [email protected] +S: Supported +F: drivers/dax/fsdev.c + DEVICE FREQUENCY (DEVFREQ) M: MyungJoo Ham <[email protected]> M: Kyungmin Park <[email protected]>
drivers/dax/Makefile +6 -0
@@ -5,10 +5,16 @@ obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o obj-$(CONFIG_DEV_DAX_CXL) += dax_cxl.o +# fsdev_dax: fs-dax compatible devdax driver (needs DEV_DAX and FS_DAX) +ifeq ($(CONFIG_FS_DAX),y) +obj-$(CONFIG_DEV_DAX) += fsdev_dax.o +endif + dax-y := super.o dax-y += bus.o device_dax-y := device.o dax_pmem-y := pmem.o dax_cxl-y := cxl.o +fsdev_dax-y := fsdev.o obj-y += hmem/
drivers/dax/bus.c +4 -0
@@ -81,6 +81,10 @@ static int dax_match_type(const struct dax_device_driver *dax_drv, struct device !IS_ENABLED(CONFIG_DEV_DAX_KMEM))return 1 ; +/* fsdev driver can also bind to device-type dax devices */ +if (dax_drv->type ==DAXDRV_FSDEV_TYPE && type ==DAXDRV_DEVICE_TYPE ) +return 1 ; +return 0 ; }
drivers/dax/bus.h +1 -0
@@ -31,6 +31,7 @@ struct dev_dax *devm_create_dev_dax(struct dev_dax_data *data);enum dax_driver_type {DAXDRV_KMEM_TYPE ,DAXDRV_DEVICE_TYPE , +DAXDRV_FSDEV_TYPE , };struct dax_device_driver {
drivers/dax/fsdev.c new file +253 -0
253 lines not on this page · raw message
fs/dax.c +1 -0
@@ -416,6 +416,7 @@ int dax_folio_reset_order(struct folio *folio)return order; } +EXPORT_SYMBOL_GPL (dax_folio_reset_order);static inline unsigned long dax_folio_put (struct folio *folio) {
include/linux/dax.h +3 -0
@@ -51,6 +51,7 @@ struct dax_holder_operations {#if IS_ENABLED (CONFIG_DAX )struct dax_device *alloc_dax (void *private,const struct dax_operations *ops); +void *dax_holder (struct dax_device *dax_dev);void put_dax (struct dax_device *dax_dev);void kill_dax (struct dax_device *dax_dev);
@@ -151,8 +152,10 @@ static inline void fs_put_dax(struct dax_device *dax_dev, void *holder)#endif /* CONFIG_BLOCK && CONFIG_FS_DAX */ #if IS_ENABLED (CONFIG_FS_DAX ) +struct dax_device *inode_dax (struct inode *inode);int dax_writeback_mapping_range (struct address_space *mapping,struct dax_device *dax_dev,struct writeback_control *wbc); +int dax_folio_reset_order (struct folio *folio);struct page *dax_layout_busy_page (struct address_space *mapping);struct page *dax_layout_busy_page_range (struct address_space *mapping,loff_t start,loff_t end);
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 4/8 dax: Save the kva from memremap
In reply to John GrovesSave the kva from memremap because we need it for iomap rw support.
Prior to famfs, there were no iomap users of /dev/dax - so the virtual address from memremap was not needed.
--- drivers/dax/dax-private.h | 2 ++ drivers/dax/fsdev.c | 1 + 2 files changed, 3 insertions(+)
drivers/dax/dax-private.h +2 -0
@@ -69,6 +69,7 @@ struct dev_dax_range {* data while the device is activated in the driver. * @region: parent region * @dax_dev: core dax functionality +* @virt_addr: kva from memremap; used by fsdev_dax * @align: alignment of this instance * @target_node: effective numa node if dev_dax memory range is onlined * @dyn_id: is this a dynamic or statically created instance
@@ -83,6 +84,7 @@ struct dev_dax_range {struct dev_dax {struct dax_region *region;struct dax_device *dax_dev; +void *virt_addr;unsigned int align;int target_node;bool dyn_id;
drivers/dax/fsdev.c +1 -0
@@ -212,6 +212,7 @@ static int fsdev_dax_probe(struct dev_dax *dev_dax)pr_debug ("%s: offset detected phys=%llx pgmap_phys=%llx offset=%llx\n" , __func__, phys, pgmap_phys, data_offset); } + dev_dax->virt_addr = addr + data_offset; inode =dax_inode (dax_dev); cdev = inode->i_cdev;
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 5/8 dax: Add dax_operations for use by fs-dax on fsdev dax
In reply to John GrovesFrom: John Groves <[email protected]>
fsdev: Add dax_operations for use by famfs.
This replicates the functionality from drivers/nvdimm/pmem.c that conventional fs-dax file systems (e.g. xfs) use to support dax read/write/mmap to a daxdev - without which famfs can't sit atop a daxdev.
- These methods are based on pmem_dax_ops from drivers/nvdimm/pmem.c
- fsdev_dax_direct_access() returns the hpa, pfn and kva. The kva was newly stored as dev_dax->virt_addr by dev_dax_probe().
- The hpa/pfn are used for mmap (dax_iomap_fault()), and the kva is used for read/write (dax_iomap_rw())
- fsdev_dax_recovery_write() and dev_dax_zero_page_range() have not been tested yet. I'm looking for suggestions as to how to test those.
- dax-private.h: add dev_dax->cached_size, which fsdev needs to remember. The dev_dax size cannot change while a driver is bound (dev_dax_resize returns -EBUSY if dev->driver is set). Caching the size at probe time allows fsdev's direct_access path can use it without acquiring dax_dev_rwsem (which isn't exported anyway).
--- drivers/dax/dax-private.h | 1 + drivers/dax/fsdev.c | 83 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+)
drivers/dax/dax-private.h +1 -0
@@ -85,6 +85,7 @@ struct dev_dax {struct dax_region *region;struct dax_device *dax_dev;void *virt_addr; +u64 cached_size;unsigned int align;int target_node;bool dyn_id;
drivers/dax/fsdev.c +83 -0
95 lines not on this page · raw message
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 6/8 dax: Add dax_set_ops() for setting dax_operations at bind time
In reply to John GrovesFrom: John Groves <[email protected]>
Add a new dax_set_ops() function that allows drivers to set the dax_operations after the dax_device has been allocated. This is needed for fsdev_dax where the operations need to be set during probe and cleared during unbind.
The fsdev driver uses devm_add_action_or_reset() for cleanup consistency, avoiding the complexity of mixing devm-managed resources with manual cleanup in a remove() callback. This ensures cleanup happens automatically in the correct reverse order when the device is unbound.
--- drivers/dax/fsdev.c | 16 ++++++++++++++++ drivers/dax/super.c | 38 +++++++++++++++++++++++++++++++++++++- include/linux/dax.h | 1 + 3 files changed, 54 insertions(+), 1 deletion(-)
drivers/dax/fsdev.c +16 -0
@@ -117,6 +117,13 @@ static void fsdev_kill(void *dev_dax)kill_dev_dax (dev_dax); } +static void fsdev_clear_ops (void *data) +{ +struct dev_dax *dev_dax = data; + +dax_set_ops (dev_dax->dax_dev,NULL ); +} +/* * Page map operations for FS-DAX mode * Similar to fsdax_pagemap_ops in drivers/nvdimm/pmem.c
@@ -310,6 +317,15 @@ static int fsdev_dax_probe(struct dev_dax *dev_dax)if (rc)return rc; +/* Set the dax operations for fs-dax access path */ + rc =dax_set_ops (dax_dev, &dev_dax_ops); +if (rc) +return rc; + + rc =devm_add_action_or_reset (dev, fsdev_clear_ops, dev_dax); +if (rc) +return rc; +run_dax (dax_dev);return devm_add_action_or_reset (dev, fsdev_kill, dev_dax); }
drivers/dax/super.c +37 -1
@@ -157,6 +157,9 @@ long dax_direct_access(struct dax_device *dax_dev, pgoff_t pgoff, long nr_pages,if (!dax_alive (dax_dev))return -ENXIO ; +if (!dax_dev->ops) +return -EOPNOTSUPP ; +if (nr_pages <0 )return -EINVAL ;
@@ -207,6 +210,10 @@ int dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff,if (!dax_alive (dax_dev))return -ENXIO ; + +if (!dax_dev->ops) +return -EOPNOTSUPP ; +/* * There are no callers that want to zero more than one page as of now. * Once users are there, this check can be removed after the
@@ -223,7 +230,7 @@ EXPORT_SYMBOL_GPL(dax_zero_page_range);size_t dax_recovery_write (struct dax_device *dax_dev,pgoff_t pgoff,void *addr,size_t bytes,struct iov_iter *iter) { -if (!dax_dev->ops->recovery_write) +if (!dax_dev->ops || !dax_dev->ops->recovery_write)return 0 ;return dax_dev->ops->recovery_write (dax_dev, pgoff, addr, bytes, iter); }
@@ -307,6 +314,35 @@ void set_dax_nomc(struct dax_device *dax_dev) }EXPORT_SYMBOL_GPL (set_dax_nomc); +/** +* dax_set_ops - set the dax_operations for a dax_device +* @dax_dev: the dax_device to configure +* @ops: the operations to set (may be NULL to clear) +* +* This allows drivers to set the dax_operations after the dax_device +* has been allocated. This is needed when the device is created before +* the driver that needs specific ops is bound (e.g., fsdev_dax binding +* to a dev_dax created by hmem). +* +* When setting non-NULL ops, fails if ops are already set (returns -EBUSY). +* When clearing ops (NULL), always succeeds. +* +* Return: 0 on success, -EBUSY if ops already set +*/ +int dax_set_ops (struct dax_device *dax_dev,const struct dax_operations *ops) +{ +if (ops) { +/* Setting ops: fail if already set */ +if (cmpxchg (&dax_dev->ops,NULL , ops) !=NULL ) +return -EBUSY ; + }else { +/* Clearing ops: always allowed */ + dax_dev->ops =NULL ; + } +return 0 ; +} +EXPORT_SYMBOL_GPL (dax_set_ops); +bool dax_alive (struct dax_device *dax_dev) {lockdep_assert_held (&dax_srcu);
include/linux/dax.h +1 -0
@@ -245,6 +245,7 @@ static inline void dax_break_layout_final(struct inode *inode)bool dax_alive (struct dax_device *dax_dev);void *dax_get_private (struct dax_device *dax_dev); +int dax_set_ops (struct dax_device *dax_dev,const struct dax_operations *ops);long dax_direct_access (struct dax_device *dax_dev,pgoff_t pgoff,long nr_pages,enum dax_access_mode mode,void **kaddr,unsigned long *pfn); size_t dax_copy_from_iter(struct dax_device *dax_dev, pgoff_t pgoff, void *addr,
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 7/8 dax: Add fs_dax_get() func to prepare dax for fs-dax usage
In reply to John GrovesThe fs_dax_get() function should be called by fs-dax file systems after opening a fsdev dax device. This adds holder_operations, which provides a memory failure callback path and effects exclusivity between callers of fs_dax_get().
fs_dax_get() is specific to fsdev_dax, so it checks the driver type (which required touching bus.[ch]). fs_dax_get() fails if fsdev_dax is not bound to the memory.
This function serves the same role as fs_dax_get_by_bdev(), which dax file systems call after opening the pmem block device.
This can't be located in fsdev.c because struct dax_device is opaque there.
This will be called by fs/fuse/famfs.c in a subsequent commit.
--- drivers/dax/bus.c | 2 -- drivers/dax/bus.h | 2 ++ drivers/dax/super.c | 66 ++++++++++++++++++++++++++++++++++++++++++++- include/linux/dax.h | 16 ++++++++--- 4 files changed, 79 insertions(+), 7 deletions(-)
drivers/dax/bus.c +0 -2
@@ -39,8 +39,6 @@ static int dax_bus_uevent(const struct device *dev, struct kobj_uevent_env *env)return add_uevent_var (env,"MODALIAS=" DAX_DEVICE_MODALIAS_FMT ,0 ); } -#define to_dax_drv (__drv) container_of_const(__drv, struct dax_device_driver, drv) -static struct dax_id *__dax_match_id (const struct dax_device_driver *dax_drv,const char *dev_name) {
drivers/dax/bus.h +2 -0
@@ -42,6 +42,8 @@ struct dax_device_driver {void (*remove)(struct dev_dax *dev); }; +#define to_dax_drv (__drv) container_of_const(__drv, struct dax_device_driver, drv) +int __dax_driver_register (struct dax_device_driver *dax_drv,struct module *module,const char *mod_name);#define dax_driver_register (driver) \
drivers/dax/super.c +65 -1
84 lines not on this page · raw message
include/linux/dax.h +12 -4
@@ -131,7 +131,6 @@ int dax_add_host(struct dax_device *dax_dev, struct gendisk *disk);void dax_remove_host (struct gendisk *disk);struct dax_device *fs_dax_get_by_bdev (struct block_device *bdev,u64 *start_off,void *holder,const struct dax_holder_operations *ops); -void fs_put_dax (struct dax_device *dax_dev,void *holder);#else static inline int dax_add_host (struct dax_device *dax_dev,struct gendisk *disk) {
@@ -146,12 +145,12 @@ static inline struct dax_device *fs_dax_get_by_bdev(struct block_device *bdev, {return NULL ; } -static inline void fs_put_dax (struct dax_device *dax_dev,void *holder) -{ -}#endif /* CONFIG_BLOCK && CONFIG_FS_DAX */ #if IS_ENABLED (CONFIG_FS_DAX ) +void fs_put_dax (struct dax_device *dax_dev,void *holder); +int fs_dax_get (struct dax_device *dax_dev,void *holder, +const struct dax_holder_operations *hops);struct dax_device *inode_dax (struct inode *inode);int dax_writeback_mapping_range (struct address_space *mapping,struct dax_device *dax_dev,struct writeback_control *wbc);
@@ -166,6 +165,15 @@ dax_entry_t dax_lock_mapping_entry(struct address_space *mapping,void dax_unlock_mapping_entry (struct address_space *mapping,unsigned long index,dax_entry_t cookie);#else +static inline void fs_put_dax (struct dax_device *dax_dev,void *holder) +{ +} + +static inline int fs_dax_get (struct dax_device *dax_dev,void *holder, +const struct dax_holder_operations *hops) +{ +return -EOPNOTSUPP ; +}static inline struct page *dax_layout_busy_page (struct address_space *mapping) {return NULL ;
Signature · 1 line
2.53.0
John Groves
maintainerPATCH v8 8/8 dax: export dax_dev_get()
In reply to John Grovesfamfs needs to look up a dax_device by dev_t when resolving fmap entries that reference character dax devices.
--- drivers/dax/super.c | 3 ++- include/linux/dax.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
drivers/dax/super.c +2 -1
@@ -521,7 +521,7 @@ static int dax_set(struct inode *inode, void *data)return 0 ; } -static struct dax_device *dax_dev_get (dev_t devt) +struct dax_device *dax_dev_get (dev_t devt) {struct dax_device *dax_dev;struct inode *inode;
@@ -544,6 +544,7 @@ static struct dax_device *dax_dev_get(dev_t devt)return dax_dev; } +EXPORT_SYMBOL_GPL (dax_dev_get);struct dax_device *alloc_dax (void *private,const struct dax_operations *ops) {
include/linux/dax.h +1 -0
@@ -55,6 +55,7 @@ struct dax_device *alloc_dax(void *private, const struct dax_operations *ops);void *dax_holder (struct dax_device *dax_dev);void put_dax (struct dax_device *dax_dev);void kill_dax (struct dax_device *dax_dev); +struct dax_device *dax_dev_get (dev_t devt);void dax_write_cache (struct dax_device *dax_dev,bool wc);bool dax_write_cache_enabled (struct dax_device *dax_dev);bool dax_synchronous (struct dax_device *dax_dev);
Signature · 1 line
2.53.0
John Groves
maintainerv8 cover famfs: port into fuse
In reply to John GrovesThis patch series on top of the prerequisite patches to dax are xavailable as a git tag at [0].
Changes v7 -> v8
Changes v7 -> v8
- Moved to inline __free declaration in fuse_get_fmap() and famfs_fuse_meta_alloc(), famfs_teardown()
- Adopted FIELD_PREP() macro rather than manual bitfield manipulation
- Minor doc edits
- I dropped adding magic numbers to include/uapi/linux/magic.h. That can be done later if appropriate
Description:
This patch series introduces famfs into the fuse file system framework. Famfs depends on the bundled dax patch set.
The famfs user space code can be found at [1].
Fuse Overview:
Famfs started as a standalone file system, but this series is intended to permanently supersede that implementation. At a high level, famfs adds two new fuse server messages:
GET_FMAP - Retrieves a famfs fmap (the file-to-dax map for a famfs
file)GET_DAXDEV - Retrieves the details of a particular daxdev that was
referenced by an fmap
Famfs Overview
Famfs exposes shared memory as a file system. Famfs consumes shared memory from dax devices, and provides memory-mappable files that map directly to the memory - no page cache involvement. Famfs differs from conventional file systems in fs-dax mode, in that it handles in-memory metadata in a sharable way (which begins with never caching dirty shared metadata).
Famfs started as a standalone file system [2,3], but the consensus at LSFMM was that it should be ported into fuse [4,5].
The key performance requirement is that famfs must resolve mapping faults without upcalls. This is achieved by fully caching the file-to-devdax metadata for all active files. This is done via two fuse client/server message/response pairs: GET_FMAP and GET_DAXDEV.
Famfs remains the first fs-dax file system that is backed by devdax rather than pmem in fs-dax mode (hence the need for the new dax mode).
Notes
- When a file is opened in a famfs mount, the OPEN is followed by a GET_FMAP message and response. The "fmap" is the full file-to-dax mapping, allowing the fuse/famfs kernel code to handle read/write/fault without any upcalls.
- After each GET_FMAP, the fmap is checked for extents that reference previously-unknown daxdevs. Each such occurrence is handled with a GET_DAXDEV message and response.
- Daxdevs are stored in a table (which might become an xarray at some point). When entries are added to the table, we acquire exclusive access to the daxdev via the fs_dax_get() call (modeled after how fs-dax handles this with pmem devices). Famfs provides holder_operations to devdax, providing a notification path in the event of memory errors or forced reconfiguration.
- If devdax notifies famfs of memory errors on a dax device, famfs currently blocks all subsequent accesses to data on that device. The recovery is to re-initialize the memory and file system. Famfs is memory, not storage...
- Because famfs uses backing (devdax) devices, only privileged mounts are supported (i.e. the fuse server requires CAP_SYS_RAWIO).
- The famfs kernel code never accesses the memory directly - it only facilitates read, write and mmap on behalf of user processes, using fmap metadata provided by its privileged fuse server. As such, the RAS of the shared memory affects applications, but not the kernel.
- Famfs has backing device(s), but they are devdax (char) rather than block. Right now there is no way to tell the vfs layer that famfs has a char backing device (unless we say it's block, but it's not). Currently we use the standard anonymous fuse fs_type - but I'm not sure that's ultimately optimal (thoughts?)
Changes v6 -> v7
- Fixed a regression in famfs_interleave_fileofs_to_daxofs() that was reported by Intel's kernel test robot
- Added a check in __fsdev_dax_direct_access() for negative return from pgoff_to_phys(), which would indicate an out-of-range offset
- Fixed a bug in __famfs_meta_free(), where not all interleaved extents were freed
- Added chunksize alignment checks in famfs_fuse_meta_alloc() and famfs_interleave_fileofs_to_daxofs() as interleaved chunks must be PTE or PMD aligned
- Simplified famfs_file_init_dax() a bit
- Re-ran CM's kernel code review prompts on the entire series and fixed several minor issues
Changes v4 -> v5 -> v6
- None. Re-sending due to technical difficulties
Changes v3 [9] -> v4
- The patch "dax: prevent driver unbind while filesystem holds device" has been dropped. Dan Williams indicated that the favored behavior is for a file system to stop working if an underlying driver is unbound, rather than preventing the unbind.
- The patch "famfs_fuse: Famfs mount opt: -o shadow=<shadowpath>" has been dropped. Found a way for the famfs user space to do without the -o opt (via getxattr).
- Squashed the fs/fuse/Kconfig patch into the first subsequent patch that needed the change
("famfs_fuse: Basic fuse kernel ABI enablement for famfs")
- Many review comments addressed.
- Addressed minor kerneldoc infractions reported by test robot.
Changes v2 [7] -> v3
- Dax: Completely new fsdev driver (drivers/dax/fsdev.c) replaces the dev_dax_iomap modifications to bus.c/device.c. Devdax devices can now be switched among 'devdax', 'famfs' and 'system-ram' modes via daxctl or sysfs.
- Dax: fsdev uses MEMORY_DEVICE_FS_DAX type and leaves folios at order-0 (no vmemmap_shift), allowing fs-dax to manage folio lifecycles dynamically like pmem does.
- Dax: The "poisoned page" problem is properly fixed via fsdev_clear_folio_state(), which clears stale mapping/compound state when fsdev binds. The temporary WARN_ON_ONCE workaround in fs/dax.c has been removed.
- Dax: Added dax_set_ops() so fsdev can set dax_operations at bind time (and clear them on unbind), since the dax_device is created before we know which driver will bind.
- Dax: Added custom bind/unbind sysfs handlers; unbind return -EBUSY if a filesystem holds the device, preventing unbind while famfs is mounted.
- Fuse: Famfs mounts now require that the fuse server/daemon has CAP_SYS_RAWIO because they expose raw memory devices.
- Fuse: Added DAX address_space_operations with noop_dirty_folio since famfs is memory-backed with no writeback required.
- Rebased to latest kernels, fully compatible with Alistair Popple et. al's recent dax refactoring.
- Ran this series through Chris Mason's code review AI prompts to check for issues - several subtle problems found and fixed.
- Dropped RFC status - this version is intended to be mergeable.
Changes v1 [8] -> v2:
- The GET_FMAP message/response has been moved from LOOKUP to OPEN, as was the pretty much unanimous consensus.
- Made the response payload to GET_FMAP variable sized (patch 12)
- Dodgy kerneldoc comments cleaned up or removed.
- Fixed memory leak of fc->shadow in patch 11 (thanks Joanne)
- Dropped many pr_debug and pr_notice calls
References
[0] - https://github.com/jagalactic/linux/tree/famfs-v8 (this patch set)
[1] - https://famfs.org (famfs user space)
[2] - https://lore.kernel.org/linux-cxl/[email protected]/
[3] - https://lore.kernel.org/linux-cxl/[email protected]/
[4] - https://lwn.net/Articles/983105/ (lsfmm 2024)
[5] - https://lwn.net/Articles/1020170/ (lsfmm 2025)
[6] - https://lore.kernel.org/linux-cxl/cover.8068ad144a7eea4a813670301f4d2a86a8e68ec4.1740713401.git-series.apopple@nvidia.com/
[7] - https://lore.kernel.org/linux-fsdevel/[email protected]/ (famfs fuse v2)
[8] - https://lore.kernel.org/linux-fsdevel/[email protected]/ (famfs fuse v1)
[9] - https://lore.kernel.org/linux-fsdevel/[email protected]/T/#mb2c868801be16eca82dab239a1d201628534aea7 (famfs fuse v3)
John Groves (10):
famfs_fuse: Update macro s/FUSE_IS_DAX/FUSE_IS_VIRTIO_DAX/
famfs_fuse: Basic fuse kernel ABI enablement for famfs
famfs_fuse: Plumb the GET_FMAP message/response
famfs_fuse: Create files with famfs fmaps
famfs_fuse: GET_DAXDEV message and daxdev_table
famfs_fuse: Plumb dax iomap and fuse read/write/mmap
famfs_fuse: Add holder_operations for dax notify_failure()
famfs_fuse: Add DAX address_space_operations with noop_dirty_folio
famfs_fuse: Add famfs fmap metadata documentation
famfs_fuse: Add documentation
Documentation/filesystems/famfs.rst | 142 ++++ Documentation/filesystems/index.rst | 1 + MAINTAINERS | 10 + fs/fuse/Kconfig | 14 + fs/fuse/Makefile | 1 + fs/fuse/dir.c | 2 +- fs/fuse/famfs.c | 1180 +++++++++++++++++++++++++++ fs/fuse/famfs_kfmap.h | 167 ++++ fs/fuse/file.c | 45 +- fs/fuse/fuse_i.h | 116 ++- fs/fuse/inode.c | 35 +- fs/fuse/iomode.c | 2 +- fs/namei.c | 1 + include/uapi/linux/fuse.h | 88 ++ 14 files changed, 1791 insertions(+), 13 deletions(-) create mode 100644 Documentation/filesystems/famfs.rst create mode 100644 fs/fuse/famfs.c create mode 100644 fs/fuse/famfs_kfmap.h
base-commit: 317638d995c2578f45c9dd0156cd509efff2a332
Signature · 1 line
2.53.0
Re: [PATCH V8 2/8] dax: Factor out dax_folio_reset_order() helper
In reply to John GrovesQuoted 15 lines · On Wed, 18 Mar 2026 20:28:20 -0500 John Groves <[email protected]> wrote:
From: John Groves <[email protected]>
Both fs/dax.c:dax_folio_put() and
drivers/dax/fsdev.c:fsdev_clear_folio_state()(the latter coming in the next commit after this one) contain nearly identical code to reset a compound DAX folio back to order-0 pages. Factor this out into a shared helper function.The new
dax_folio_reset_order()function:
- Clears the folio's mapping and share count
- Resets compound folio state viafolio_reset_order()
- Clears PageHead andcompound_headfor each sub-page
- Restores the pgmap pointer for each resulting order-0 folio
- Returns the original folio order (for callers that need to advance by that many pages)
Quoted 7 lines
This simplifies
fsdev_clear_folio_state()from ~50 lines to ~15 lines while maintaining the same functionality in both call sites.Suggested-by Jonathan Cameron <[email protected]>Reviewed-by Ira Weiny <[email protected]>Reviewed-by Dave Jiang <[email protected]>Signed-off-by John Groves <[email protected]>
Comment below. I may well be needing more coffee, or failing wrt to background knowledge as I only occasionally dip into dax.
Quoted 19 lines
--- fs/dax.c | 60 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 42 insertions(+), 18 deletions(-)fs/dax.c +7
-0@@ -378,6 +378,45 @@ static void dax_folio_make_shared(struct folio *folio) folio->share =1 ; } +/** +* dax_folio_reset_order - Reset a compound DAX folio to order-0 pages +* @folio: The folio to reset +* +* Splits a compound folio back into individual order-0 pages, +* clearing compound state and restoring pgmap pointers. +*
fs/dax.c static void dax_folio_make_shared(struct folio *folio) · line 397
Quoted 10 lines
fs/dax.c +10
-0+* Returns: the original folio order (0 if already order-0) +*/ +int dax_folio_reset_order (struct folio *folio) +{ +struct dev_pagemap *pgmap =page_pgmap (&folio->page); +int order =folio_order (folio); +int i; + + folio->mapping =NULL ; + folio->share =0 ;
This is different from the code you are replacing..
Just above the call to this in dax_folio_put()
if (!dax_folio_is_shared(folio))
// in here is the interesting bit...
ref = 0; else //this is fine because either it's still > 0 and we return //or it is zero and you are writing that again. ref = --folio->share; if (ref) return ref;
So the path that bothers me is if !dax_folio_is_shared() can return false with shared != 0
/*
* A DAX folio is considered shared if it has no mapping set and ->share (which
* shares the ->index field) is non-zero. Note this may return false even if the
* page is shared between multiple files but has not yet actually been mapped
* into multiple address spaces.
*/
static inline bool dax_folio_is_shared(struct folio *folio)
{
return !folio->mapping && folio->share;
}So it can if !folio->mapping is false (i.e. folio->mapping is set)
Now I have zero idea of whether this is a real path and have
a long review queue so not looking into it for now.
However if it's not then I'd expect some commentary in the patch description to say why it's not a problem. Maybe even a precursor patch adding the folio->share so there is a place to state clearly that it doesn't matter and why.
fs/dax.c static void dax_folio_make_shared(struct folio *folio) · line 400
Quoted 3 lines
fs/dax.c +3
-0+ +if (!order) { + folio->pgmap = pgmap;
This is also different...
fs/dax.c static void dax_folio_make_shared(struct folio *folio) · line 406
Quoted 6 lines
fs/dax.c +6
-0+return 0 ; + } + +folio_reset_order (folio); + +for (i =0 ; i < (1UL << order); i++) {
I'd take advantage of evolving conventions and do
for (int i = 0; i < ...)
Quoted 48 lines
fs/dax.c +16
-18+struct page *page =folio_page (folio, i); +struct folio *f = (struct folio *)page; + +ClearPageHead (page); +clear_compound_head (page); + f->mapping =NULL ; + f->share =0 ; + f->pgmap = pgmap; + } + +return order; +} +static inline unsigned long dax_folio_put (struct folio *folio) {unsigned long ref;@@ -391,28 +430,13 @@ static inline unsigned long dax_folio_put(struct folio *folio)if (ref)return ref; - folio->mapping =NULL ; - order =folio_order (folio); -if (!order) -return 0 ; -folio_reset_order (folio); + order =dax_folio_reset_order (folio); +/* Debug check: verify refcounts are zero for all sub-folios */ for (i =0 ; i < (1UL << order); i++) { -struct dev_pagemap *pgmap =page_pgmap (&folio->page);struct page *page =folio_page (folio, i); -struct folio *new_folio = (struct folio *)page; -ClearPageHead (page); -clear_compound_head (page); - - new_folio->mapping =NULL ; -/* -* Reset pgmap which was over-written by -* prep_compound_page(). -*/ - new_folio->pgmap = pgmap; - new_folio->share =0 ; -WARN_ON_ONCE (folio_ref_count (new_folio)); +WARN_ON_ONCE (folio_ref_count ((struct folio *)page)); }return ref;
Re: [PATCH V8 3/8] dax: add fsdev.c driver for fs-dax on character dax
In reply to John GrovesQuoted 19 lines · On Wed, 18 Mar 2026 20:28:37 -0500 John Groves <[email protected]> wrote:
The new fsdev driver provides pages/folios initialized compatibly with fsdax - normal rather than devdax-style refcounting, and starting out with order-0 folios.
When fsdev binds to a daxdev, it is usually (always?) switching from the devdax mode (
device.c), which pre-initializes compound folios according to its alignment. Fsdev usesfsdev_clear_folio_state()to switch the folios into a fsdax-compatible state.A side effect of this is that raw mmap doesn't (can't?) work on an fsdev dax instance. Accordingly, The fsdev driver does not provide raw mmap - devices must be put in 'devdax' mode (
drivers/dax/device.c) to get raw mmap capability.In this commit is just the framework, which remaps pages/folios compatibly with fsdax.
Enabling dax changes:
Quoted 7 lines
-
bus.h: addDAXDRV_FSDEV_TYPEdriver type
-bus.c: allowDAXDRV_FSDEV_TYPEdrivers to bind to daxdevs
-dax.h: prototypeinode_dax(), which fsdev needsSuggested-by Dan Williams <[email protected]>Suggested-by Gregory Price <[email protected]>Signed-off-by John Groves <[email protected]>
A few comments inline. I think some of the code here could be moved to a helper library used by both this and device.c
Quoted 35 lines
--- MAINTAINERS | 8 ++ drivers/dax/Makefile | 6 + drivers/dax/bus.c | 4 + drivers/dax/bus.h | 1 + drivers/dax/fsdev.c | 253 +++++++++++++++++++++++++++++++++++++++++++ fs/dax.c | 1 + include/linux/dax.h | 3 + 7 files changed, 276 insertions(+) create mode 100644 drivers/dax/fsdev.cMAINTAINERS +8
-0@@ -7298,6 +7298,14 @@ L: [email protected] S: Supported F: drivers/dax/ +DEVICE DIRECT ACCESS (DAX) [fsdev_dax] +M: John Groves <[email protected]> +M: John Groves <[email protected]> +L: [email protected] +L: [email protected] +S: Supported +F: drivers/dax/fsdev.c + DEVICE FREQUENCY (DEVFREQ) M: MyungJoo Ham <[email protected]> M: Kyungmin Park <[email protected]>drivers/dax/Makefile +0
-0
drivers/dax/Makefile obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o · line 11
Quoted 7 lines
drivers/dax/Makefile +4
-0@@ -5,10 +5,16 @@ obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o obj-$(CONFIG_DEV_DAX_CXL) += dax_cxl.o +# fsdev_dax: fs-dax compatible devdax driver (needs DEV_DAX and FS_DAX) +ifeq ($(CONFIG_FS_DAX),y) +obj-$(CONFIG_DEV_DAX) += fsdev_dax.o +endif
Why not throw in a new CONFIG_FSDAX_DEV and handle the dependencies in Kconfig?
Quoted 15 lines
drivers/dax/Makefile +2
-0+ dax-y := super.o dax-y += bus.o device_dax-y := device.o dax_pmem-y := pmem.o dax_cxl-y := cxl.o +fsdev_dax-y := fsdev.o obj-y += hmem/drivers/dax/fsdev.c new file +0
-0
drivers/dax/fsdev.c line 124
Quoted 7 lines
drivers/dax/fsdev.c new file +6
-0+static int fsdev_dax_probe (struct dev_dax *dev_dax) +{ +struct dax_device *dax_dev = dev_dax->dax_dev; +struct device *dev = &dev_dax->dev; +struct dev_pagemap *pgmap; +u64 data_offset =0 ;
See below. I think you can useful reduce scope of this one.
drivers/dax/fsdev.c line 129
Quoted 5 lines
drivers/dax/fsdev.c new file +5
-0+struct inode *inode; +struct cdev *cdev; +void *addr; +int rc, i; +
There is a lot of duplication in here with dax/device.c Is any of it suitable for shared helpers?
Quoted 7 lines
drivers/dax/fsdev.c new file +7
-0+if (static_dev_dax (dev_dax)) { +if (dev_dax->nr_range >1 ) { +dev_warn (dev,"static pgmap / multi-range device conflict\n" ); +return -EINVAL ; + } + + pgmap = dev_dax->pgmap;
drivers/dax/fsdev.c line 146
Quoted 10 lines
drivers/dax/fsdev.c new file +10
-0+ }else { +size_t pgmap_size; + +if (dev_dax->pgmap) { +dev_warn (dev,"dynamic-dax with pre-populated page map\n" ); +return -EINVAL ; + } + + pgmap_size =struct_size (pgmap, ranges, dev_dax->nr_range -1 ); + pgmap =devm_kzalloc (dev, pgmap_size,GFP_KERNEL );
Bonus space before GFP_KERNEL.
Quoted 13 lines
drivers/dax/fsdev.c new file +13
-0+if (!pgmap) +return -ENOMEM ; + + pgmap->nr_range = dev_dax->nr_range; + dev_dax->pgmap = pgmap; + +for (i =0 ; i < dev_dax->nr_range; i++) { +struct range *range = &dev_dax->ranges[i].range; + + pgmap->ranges[i] = *range; + } + } +
drivers/dax/fsdev.c line 169
Quoted 10 lines
drivers/dax/fsdev.c new file +10
-0+for (i =0 ; i < dev_dax->nr_range; i++) { +struct range *range = &dev_dax->ranges[i].range; + +if (!devm_request_mem_region (dev, range->start, +range_len (range),dev_name (dev))) { +dev_warn (dev,"mapping%d: %#llx-%#llx could not reserve range\n" , + i, range->start, range->end); +return -EBUSY ; + } + }
Everything above here is shared. Some sort of _init() or similar library function seems in order.
Quoted 34 lines
drivers/dax/fsdev.c new file +34
-0+ +/* +* FS-DAX compatible mode: Use MEMORY_DEVICE_FS_DAX type and +* do NOT set vmemmap_shift. This leaves folios at order-0, +* allowing fs-dax to dynamically create compound folios as needed +* (similar to pmem behavior). +*/ + pgmap->type =MEMORY_DEVICE_FS_DAX ; + pgmap->ops = &fsdev_pagemap_ops; + pgmap->owner = dev_dax; + +/* +* CRITICAL DIFFERENCE from device.c: +* We do NOT set vmemmap_shift here, even if align > PAGE_SIZE. +* This ensures folios remain order-0 and are compatible with +* fs-dax's folio management. +*/ + + addr =devm_memremap_pages (dev, pgmap); +if (IS_ERR (addr)) +return PTR_ERR (addr); + +/* +* Clear any stale compound folio state left over from a previous +* driver (e.g., device_dax with vmemmap_shift). Also register this +* as a devm action so folio state is cleared on unbind, ensuring +* clean pages for subsequent drivers (e.g., kmem for system-ram). +*/ +fsdev_clear_folio_state (dev_dax); + rc =devm_add_action_or_reset (dev, fsdev_clear_folio_state_action, + dev_dax); +if (rc) +return rc; +
drivers/dax/fsdev.c line 213
Quoted 10 lines
drivers/dax/fsdev.c new file +10
-0+/* Detect whether the data is at a non-zero offset into the memory */ +if (pgmap->range.start != dev_dax->ranges[0 ].range.start) { +u64 phys = dev_dax->ranges[0 ].range.start; +u64 pgmap_phys = dev_dax->pgmap[0 ].range.start; + +if (!WARN_ON (pgmap_phys > phys)) + data_offset = phys - pgmap_phys; + +pr_debug ("%s: offset detected phys=%llx pgmap_phys=%llx offset=%llx\n" , + __func__, phys, pgmap_phys, data_offset);
Might change later, but at least at this point you could pull declaration of data_offset into this scope.
Quoted 24 lines
drivers/dax/fsdev.c new file +18
-0+ } + + inode =dax_inode (dax_dev); + cdev = inode->i_cdev; +cdev_init (cdev, &fsdev_fops); + cdev->owner = dev->driver->owner; +cdev_set_parent (cdev, &dev->kobj); + rc =cdev_add (cdev, dev->devt,1 ); +if (rc) +return rc; + + rc =devm_add_action_or_reset (dev, fsdev_cdev_del, cdev); +if (rc) +return rc; + +run_dax (dax_dev); +return devm_add_action_or_reset (dev, fsdev_kill, dev_dax); +}include/linux/dax.h +0
-0
include/linux/dax.h struct dax_holder_operations { · line 54
Quoted 4 lines
include/linux/dax.h +1
-0@@ -51,6 +51,7 @@ struct dax_holder_operations {#if IS_ENABLED (CONFIG_DAX )struct dax_device *alloc_dax (void *private,const struct dax_operations *ops); +
Unrelated change. Tidy this up for v9.
include/linux/dax.h static inline void fs_put_dax(struct dax_device *dax_dev, void *holder) · line 155
Quoted 8 lines
include/linux/dax.h +1
-0void *dax_holder (struct dax_device *dax_dev);void put_dax (struct dax_device *dax_dev);void kill_dax (struct dax_device *dax_dev);@@ -151,8 +152,10 @@ static inline void fs_put_dax(struct dax_device *dax_dev, void *holder)#endif /* CONFIG_BLOCK && CONFIG_FS_DAX */ #if IS_ENABLED (CONFIG_FS_DAX ) +struct dax_device *inode_dax (struct inode *inode);
Already in dax_private.h so why does it want to be here?
Quoted 6 lines
include/linux/dax.h +1
-0int dax_writeback_mapping_range (struct address_space *mapping,struct dax_device *dax_dev,struct writeback_control *wbc); +int dax_folio_reset_order (struct folio *folio);struct page *dax_layout_busy_page (struct address_space *mapping);struct page *dax_layout_busy_page_range (struct address_space *mapping,loff_t start,loff_t end);
PATCH v8 1/10 famfs_fuse: Update macro s/FUSE_IS_DAX/FUSE_IS_VIRTIO_DAX/
In reply to John GrovesFrom: John Groves <[email protected]>
Virtio_fs now needs to determine if an inode is DAX && not famfs. This relaces the FUSE_IS_DAX() macro with FUSE_IS_VIRTIO_DAX(), in preparation for famfs in later commits. The dummy fuse_file_famfs() macro will be replaced with a working function.
--- fs/fuse/dir.c | 2 +- fs/fuse/file.c | 13 ++++++++----- fs/fuse/fuse_i.h | 9 ++++++++- fs/fuse/inode.c | 4 ++-- fs/fuse/iomode.c | 2 +- 5 files changed, 20 insertions(+), 10 deletions(-)
fs/fuse/dir.c +1 -1
@@ -2161,7 +2161,7 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, is_truncate = true; } -if (FUSE_IS_DAX (inode) && is_truncate) { +if (FUSE_IS_VIRTIO_DAX (fi) && is_truncate) {filemap_invalidate_lock (mapping); fault_blocked = true; err =fuse_dax_break_layouts (inode,0 ,-1 );
fs/fuse/file.c +8 -5
@@ -252,7 +252,7 @@ static int fuse_open(struct inode *inode, struct file *file)int err;bool is_truncate = (file->f_flags &O_TRUNC ) && fc->atomic_o_trunc;bool is_wb_truncate = is_truncate && fc->writeback_cache; -bool dax_truncate = is_truncate &&FUSE_IS_DAX (inode); +bool dax_truncate = is_truncate &&FUSE_IS_VIRTIO_DAX (fi);if (fuse_is_bad (inode))return -EIO ;
@@ -1812,11 +1812,12 @@ static ssize_t fuse_file_read_iter(struct kiocb *iocb, struct iov_iter *to)struct file *file = iocb->ki_filp;struct fuse_file *ff = file->private_data;struct inode *inode =file_inode (file); +struct fuse_inode *fi =get_fuse_inode (inode);if (fuse_is_bad (inode))return -EIO ; -if (FUSE_IS_DAX (inode)) +if (FUSE_IS_VIRTIO_DAX (fi))return fuse_dax_read_iter (iocb, to);/* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */
@@ -1833,11 +1834,12 @@ static ssize_t fuse_file_write_iter(struct kiocb *iocb, struct iov_iter *from)struct file *file = iocb->ki_filp;struct fuse_file *ff = file->private_data;struct inode *inode =file_inode (file); +struct fuse_inode *fi =get_fuse_inode (inode);if (fuse_is_bad (inode))return -EIO ; -if (FUSE_IS_DAX (inode)) +if (FUSE_IS_VIRTIO_DAX (fi))return fuse_dax_write_iter (iocb, from);/* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */
@@ -2370,10 +2372,11 @@ static int fuse_file_mmap(struct file *file, struct vm_area_struct *vma)struct fuse_file *ff = file->private_data;struct fuse_conn *fc = ff->fm->fc;struct inode *inode =file_inode (file); +struct fuse_inode *fi =get_fuse_inode (inode);int rc;/* DAX mmap is superior to direct_io mmap */ -if (FUSE_IS_DAX (inode)) +if (FUSE_IS_VIRTIO_DAX (fi))return fuse_dax_mmap (file, vma);/*
@@ -2934,7 +2937,7 @@ static long fuse_file_fallocate(struct file *file, int mode, loff_t offset, .mode = mode };int err; -bool block_faults =FUSE_IS_DAX (inode) && +bool block_faults =FUSE_IS_VIRTIO_DAX (fi) && (!(mode &FALLOC_FL_KEEP_SIZE ) || (mode & (FALLOC_FL_PUNCH_HOLE |FALLOC_FL_ZERO_RANGE )));
fs/fuse/fuse_i.h +8 -1
@@ -1508,7 +1508,14 @@ void fuse_free_conn(struct fuse_conn *fc);/* dax.c */ -#define FUSE_IS_DAX (inode) (IS_ENABLED(CONFIG_FUSE_DAX) && IS_DAX(inode)) +static inline bool fuse_file_famfs (struct fuse_inode *fuse_inode)/* Will be superseded */ +{ + (void )fuse_inode; +return false; +} +#define FUSE_IS_VIRTIO_DAX (fuse_inode) (IS_ENABLED(CONFIG_FUSE_DAX) \ + && IS_DAX(&fuse_inode->inode) \ + && !fuse_file_famfs(fuse_inode))ssize_t fuse_dax_read_iter (struct kiocb *iocb,struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);
fs/fuse/inode.c +2 -2
@@ -162,7 +162,7 @@ static void fuse_evict_inode(struct inode *inode)/* Will write inode on close/munmap and in all other dirtiers */ WARN_ON (inode_state_read_once (inode) &I_DIRTY_INODE ); -if (FUSE_IS_DAX (inode)) +if (FUSE_IS_VIRTIO_DAX (fi))dax_break_layout_final (inode);truncate_inode_pages_final (&inode->i_data);
@@ -170,7 +170,7 @@ static void fuse_evict_inode(struct inode *inode)if (inode->i_sb->s_flags &SB_ACTIVE ) {struct fuse_conn *fc =get_fuse_conn (inode); -if (FUSE_IS_DAX (inode)) +if (FUSE_IS_VIRTIO_DAX (fi))fuse_dax_inode_cleanup (inode);if (fi->nlookup) { fuse_queue_forget(fc, fi->forget, fi->nodeid,
fs/fuse/iomode.c +1 -1
@@ -203,7 +203,7 @@ int fuse_file_io_open(struct file *file, struct inode *inode)* io modes are not relevant with DAX and with server that does not * implement open. */ -if (FUSE_IS_DAX (inode) || !ff->args) +if (FUSE_IS_VIRTIO_DAX (fi) || !ff->args)return 0 ;/*
Signature · 1 line
2.53.0
PATCH v8 2/10 famfs_fuse: Basic fuse kernel ABI enablement for famfs
In reply to John GrovesFrom: John Groves <[email protected]>
This patch starts the kernel ABI enablement of famfs in fuse.
- Kconfig: Add FUSE_FAMFS_DAX config parameter, to control compilation of famfs within fuse.
- FUSE_DAX_FMAP flag in INIT request/reply
- fuse_conn->famfs_iomap (enable famfs-mapped files) to denote a famfs-enabled connection
--- fs/fuse/Kconfig | 14 ++++++++++++++ fs/fuse/fuse_i.h | 3 +++ fs/fuse/inode.c | 6 ++++++ include/uapi/linux/fuse.h | 5 +++++ 4 files changed, 28 insertions(+)
fs/fuse/Kconfig +14 -0
@@ -76,3 +76,17 @@ config FUSE_IO_URING If you want to allow fuse server/client communication through io-uring, answer Y + +config FUSE_FAMFS_DAX + bool "FUSE support for fs-dax filesystems backed by devdax" + depends on FUSE_FS + depends on DEV_DAX + depends on FS_DAX + default FUSE_FS + help + This enables the fabric-attached memory file system (famfs), + which enables formatting devdax memory as a file system. Famfs + is primarily intended for scale-out shared access to + disaggregated memory. + + To enable famfs or other fuse/fs-dax file systems, answer Y
fs/fuse/fuse_i.h +3 -0
@@ -921,6 +921,9 @@ struct fuse_conn {/* Is synchronous FUSE_INIT allowed? */ unsigned int sync_init:1; +/* dev_dax_iomap support for famfs */ + unsigned int famfs_iomap:1; +/* Use io_uring for communication */ unsigned int io_uring;
fs/fuse/inode.c +6 -0
@@ -1456,6 +1456,10 @@ static void process_init_reply(struct fuse_mount *fm, struct fuse_args *args,if (flags &FUSE_REQUEST_TIMEOUT ) timeout = arg->request_timeout; + +if (IS_ENABLED (CONFIG_FUSE_FAMFS_DAX ) && + flags &FUSE_DAX_FMAP ) + fc->famfs_iomap =1 ; }else { ra_pages = fc->max_read /PAGE_SIZE ; fc->no_lock =1 ;
@@ -1517,6 +1521,8 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm) flags |=FUSE_SUBMOUNTS ;if (IS_ENABLED (CONFIG_FUSE_PASSTHROUGH )) flags |=FUSE_PASSTHROUGH ; +if (IS_ENABLED (CONFIG_FUSE_FAMFS_DAX )) + flags |=FUSE_DAX_FMAP ;/* * This is just an information flag for fuse server. No need to check
include/uapi/linux/fuse.h +5 -0
@@ -240,6 +240,9 @@* - add FUSE_COPY_FILE_RANGE_64 * - add struct fuse_copy_file_range_out * - add FUSE_NOTIFY_PRUNE +* +* 7.46 +* - Add FUSE_DAX_FMAP capability - ability to handle in-kernel fsdax maps */ #ifndef _LINUX_FUSE_H
@@ -448,6 +451,7 @@ struct fuse_file_lock {* FUSE_OVER_IO_URING: Indicate that client supports io-uring * FUSE_REQUEST_TIMEOUT: kernel supports timing out requests. * init_out.request_timeout contains the timeout (in secs) +* FUSE_DAX_FMAP: kernel supports dev_dax_iomap (aka famfs) fmaps */ #define FUSE_ASYNC_READ (1 << 0)#define FUSE_POSIX_LOCKS (1 << 1)
@@ -495,6 +499,7 @@ struct fuse_file_lock {#define FUSE_ALLOW_IDMAP (1ULL << 40)#define FUSE_OVER_IO_URING (1ULL << 41)#define FUSE_REQUEST_TIMEOUT (1ULL << 42) +#define FUSE_DAX_FMAP (1ULL << 43)/** * CUSE INIT request/reply flags
Signature · 1 line
2.53.0
PATCH v8 3/10 famfs_fuse: Plumb the GET_FMAP message/response
In reply to John GrovesFrom: John Groves <[email protected]>
Upon completion of an OPEN, if we're in famfs-mode we do a GET_FMAP to retrieve and cache up the file-to-dax map in the kernel. If this succeeds, read/write/mmap are resolved direct-to-dax with no upcalls.
--- MAINTAINERS | 8 +++++ fs/fuse/Makefile | 1 + fs/fuse/famfs.c | 73 +++++++++++++++++++++++++++++++++++++++ fs/fuse/file.c | 14 +++++++- fs/fuse/fuse_i.h | 70 ++++++++++++++++++++++++++++++++++--- fs/fuse/inode.c | 8 ++++- fs/fuse/iomode.c | 2 +- include/uapi/linux/fuse.h | 7 ++++ 8 files changed, 175 insertions(+), 8 deletions(-) create mode 100644 fs/fuse/famfs.c
MAINTAINERS +8 -0
@@ -10510,6 +10510,14 @@ F: fs/fuse/ F: include/uapi/linux/fuse.h F: tools/testing/selftests/filesystems/fuse/ +FUSE [FAMFS Fabric-Attached Memory File System] +M: John Groves <[email protected]> +M: John Groves <[email protected]> +L: [email protected] +L: [email protected] +S: Supported +F: fs/fuse/famfs.c + FUTEX SUBSYSTEM M: Thomas Gleixner <[email protected]> M: Ingo Molnar <[email protected]>
fs/fuse/Makefile +1 -0
@@ -17,5 +17,6 @@ fuse-$(CONFIG_FUSE_DAX) += dax.o fuse-$(CONFIG_FUSE_PASSTHROUGH) += passthrough.o backing.o fuse-$(CONFIG_SYSCTL) += sysctl.o fuse-$(CONFIG_FUSE_IO_URING) += dev_uring.o +fuse-$(CONFIG_FUSE_FAMFS_DAX) += famfs.o virtiofs-y := virtio_fs.o
fs/fuse/famfs.c new file +73 -0
73 lines not on this page · raw message
fs/fuse/file.c +13 -1
@@ -277,6 +277,16 @@ static int fuse_open(struct inode *inode, struct file *file) err =fuse_do_open (fm,get_node_id (inode), file, false);if (!err) { ff = file->private_data; + +if ((fm->fc->famfs_iomap) && (S_ISREG (inode->i_mode))) { +/* Get the famfs fmap - failure is fatal */ + err =fuse_get_fmap (fm, inode); +if (err) { +fuse_sync_release (fi, ff, file->f_flags); + goto out_nowrite; + } + } + err =fuse_finish_open (inode, file);if (err)fuse_sync_release (fi, ff, file->f_flags);
@@ -284,12 +294,14 @@ static int fuse_open(struct inode *inode, struct file *file)fuse_truncate_update_attr (inode, file); } +out_nowrite:if (is_wb_truncate || dax_truncate)fuse_release_nowrite (inode);if (!err) {if (is_truncate)truncate_pagecache (inode,0 ); -else if (!(ff->open_flags &FOPEN_KEEP_CACHE )) +else if (!(ff->open_flags &FOPEN_KEEP_CACHE ) && + !fuse_file_famfs (fi))invalidate_inode_pages2 (inode->i_mapping); }if (dax_truncate)
fs/fuse/fuse_i.h +65 -5
86 lines not on this page · raw message
fs/fuse/inode.c +7 -1
@@ -120,6 +120,9 @@ static struct inode *fuse_alloc_inode(struct super_block *sb)if (IS_ENABLED (CONFIG_FUSE_PASSTHROUGH ))fuse_inode_backing_set (fi,NULL ); +if (IS_ENABLED (CONFIG_FUSE_FAMFS_DAX )) +famfs_meta_set (fi,NULL ); +return &fi->inode; out_free_forget:
@@ -141,6 +144,9 @@ static void fuse_free_inode(struct inode *inode)if (IS_ENABLED (CONFIG_FUSE_PASSTHROUGH ))fuse_backing_put (fuse_inode_backing (fi)); +if (S_ISREG (inode->i_mode) &&fuse_file_famfs (fi)) +famfs_meta_free (fi); +kmem_cache_free (fuse_inode_cachep, fi); }
@@ -162,7 +168,7 @@ static void fuse_evict_inode(struct inode *inode)/* Will write inode on close/munmap and in all other dirtiers */ WARN_ON (inode_state_read_once (inode) &I_DIRTY_INODE ); -if (FUSE_IS_VIRTIO_DAX (fi)) +if (FUSE_IS_VIRTIO_DAX (fi) ||fuse_file_famfs (fi))dax_break_layout_final (inode);truncate_inode_pages_final (&inode->i_data);
fs/fuse/iomode.c +1 -1
@@ -203,7 +203,7 @@ int fuse_file_io_open(struct file *file, struct inode *inode)* io modes are not relevant with DAX and with server that does not * implement open. */ -if (FUSE_IS_VIRTIO_DAX (fi) || !ff->args) +if (FUSE_IS_VIRTIO_DAX (fi) ||fuse_file_famfs (fi) || !ff->args)return 0 ;/*
include/uapi/linux/fuse.h +7 -0
@@ -669,6 +669,9 @@ enum fuse_opcode {FUSE_STATX =52 ,FUSE_COPY_FILE_RANGE_64 =53 , +/* Famfs / devdax opcodes */ +FUSE_GET_FMAP =54 , +/* CUSE specific operations */ CUSE_INIT = 4096,
@@ -1313,4 +1316,8 @@ struct fuse_uring_cmd_req { uint8_t padding[6]; }; +/* Famfs fmap message components */ + +#define FAMFS_FMAP_MAX 32768/* Largest supported fmap message */ +#endif /* _LINUX_FUSE_H */
Signature · 1 line
2.53.0
PATCH v8 4/10 famfs_fuse: Create files with famfs fmaps
In reply to John GrovesFrom: John Groves <[email protected]>
On completion of GET_FMAP message/response, setup the full famfs metadata such that it's possible to handle read/write/mmap directly to dax. Note that the devdax_iomap plumbing is not in yet...
* Add famfs_kfmap.h: in-memory structures for resolving famfs file maps (fmaps) to dax.
* famfs.c: allocate, initialize and free fmaps
* inode.c: only allow famfs mode if the fuse server has CAP_SYS_RAWIO
* Update MAINTAINERS for the new file.
--- MAINTAINERS | 1 + fs/fuse/famfs.c | 339 +++++++++++++++++++++++++++++++++++++- fs/fuse/famfs_kfmap.h | 67 ++++++++ fs/fuse/fuse_i.h | 8 +- fs/fuse/inode.c | 20 ++- include/uapi/linux/fuse.h | 56 +++++++ 6 files changed, 481 insertions(+), 10 deletions(-) create mode 100644 fs/fuse/famfs_kfmap.h
MAINTAINERS +1 -0
@@ -10517,6 +10517,7 @@ L: [email protected] L: [email protected] S: Supported F: fs/fuse/famfs.c +F: fs/fuse/famfs_kfmap.h FUTEX SUBSYSTEM M: Thomas Gleixner <[email protected]>
fs/fuse/famfs.c +333 -6
353 lines not on this page · raw message
fs/fuse/famfs_kfmap.h new file +67 -0
67 lines not on this page · raw message
fs/fuse/fuse_i.h +7 -1
@@ -1642,6 +1642,9 @@ extern void fuse_sysctl_unregister(void);/* famfs.c */ #if IS_ENABLED (CONFIG_FUSE_FAMFS_DAX ) +int famfs_file_init_dax (struct fuse_mount *fm, +struct inode *inode,void *fmap_buf, +size_t fmap_size);void __famfs_meta_free (void *map);/* Set fi->famfs_meta = NULL regardless of prior value */
@@ -1659,7 +1662,10 @@ static inline struct fuse_backing *famfs_meta_set(struct fuse_inode *fi,static inline void famfs_meta_free (struct fuse_inode *fi) { -famfs_meta_set (fi,NULL ); +if (fi->famfs_meta !=NULL ) { +__famfs_meta_free (fi->famfs_meta); +famfs_meta_set (fi,NULL ); + } }static inline int fuse_file_famfs (struct fuse_inode *fi)
fs/fuse/inode.c +17 -3
@@ -10,6 +10,7 @@#include "fuse_dev_i.h" #include "dev_uring_i.h" +#include <linux/bitfield.h> #include <linux/dax.h> #include <linux/pagemap.h> #include <linux/slab.h>
@@ -1464,8 +1465,21 @@ static void process_init_reply(struct fuse_mount *fm, struct fuse_args *args, timeout = arg->request_timeout;if (IS_ENABLED (CONFIG_FUSE_FAMFS_DAX ) && - flags &FUSE_DAX_FMAP ) - fc->famfs_iomap =1 ; + flags &FUSE_DAX_FMAP ) { +/* famfs_iomap is only allowed if the fuse +* server has CAP_SYS_RAWIO. This was checked +* in fuse_send_init, and FUSE_DAX_IOMAP was +* set in in_flags if so. Only allow enablement +* if we find it there. This function is +* normally not running in fuse server context, +* so we can't do the capability check here... +*/ +u64 in_flags =FIELD_PREP (GENMASK_ULL (63 ,32 ), ia->in.flags2) + | ia->in.flags; + +if (in_flags &FUSE_DAX_FMAP ) + fc->famfs_iomap =1 ; + } }else { ra_pages = fc->max_read /PAGE_SIZE ; fc->no_lock =1 ;
@@ -1527,7 +1541,7 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm) flags |=FUSE_SUBMOUNTS ;if (IS_ENABLED (CONFIG_FUSE_PASSTHROUGH )) flags |=FUSE_PASSTHROUGH ; -if (IS_ENABLED (CONFIG_FUSE_FAMFS_DAX )) +if (IS_ENABLED (CONFIG_FUSE_FAMFS_DAX ) &&capable (CAP_SYS_RAWIO )) flags |=FUSE_DAX_FMAP ;/*
include/uapi/linux/fuse.h +56 -0
68 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v8 5/10 famfs_fuse: GET_DAXDEV message and daxdev_table
In reply to John GrovesFrom: John Groves <[email protected]>
- The new GET_DAXDEV message/response is added
- The famfs.c:famfs_teardown() function is added as a primary teardown function for famfs.
- The command it triggered by the update_daxdev_table() call, if there are any daxdevs in the subject fmap that are not represented in the daxdev_table yet.
- fs/namei.c: export may_open_dev()
--- fs/fuse/famfs.c | 227 +++++++++++++++++++++++++++++++++++++- fs/fuse/famfs_kfmap.h | 26 +++++ fs/fuse/fuse_i.h | 19 ++++ fs/fuse/inode.c | 7 +- fs/namei.c | 1 + include/uapi/linux/fuse.h | 20 ++++ 6 files changed, 298 insertions(+), 2 deletions(-)
fs/fuse/famfs.c +226 -1
@@ -21,6 +21,228 @@#include "famfs_kfmap.h" #include "fuse_i.h" +/* +* famfs_teardown() +* +* Deallocate famfs metadata for a fuse_conn +*/ +void +famfs_teardown (struct fuse_conn *fc) +{ + struct famfs_dax_devlist *devlist __free(kfree) = fc->dax_devlist; +int i; + + fc->dax_devlist =NULL ; + +if (!devlist) +return ; + +if (!devlist->devlist) +return ; + +/* Close & release all the daxdevs in our table */ +for (i =0 ; i < devlist->nslots; i++) { +struct famfs_daxdev *dd = &devlist->devlist[i]; + +if (!dd->valid) +continue ; + +/* Release reference from dax_dev_get() */ +if (dd->devp) +put_dax (dd->devp); + +kfree (dd->name); + } +kfree (devlist->devlist); +} + +static int +famfs_verify_daxdev (const char *pathname,dev_t *devno) +{ +struct inode *inode; +struct path path; +int err; + +if (!pathname || !*pathname) +return -EINVAL ; + + err =kern_path (pathname,LOOKUP_FOLLOW , &path); +if (err) +return err; + + inode =d_backing_inode (path.dentry); +if (!S_ISCHR (inode->i_mode)) { + err = -EINVAL ; + goto out_path_put; + } + +if (!may_open_dev (&path)) {/* had to export this */ + err = -EACCES ; + goto out_path_put; + } + + *devno = inode->i_rdev; + +out_path_put: +path_put (&path); +return err; +} + +/** +* famfs_fuse_get_daxdev() - Retrieve info for a DAX device from fuse server +* +* Send a GET_DAXDEV message to the fuse server to retrieve info on a +* dax device. +* +* @fm: fuse_mount +* @index: the index of the dax device; daxdevs are referred to by index +* in fmaps, and the server resolves the index to a particular daxdev +* +* Returns: 0=success +* -errno=failure +*/ +static int +famfs_fuse_get_daxdev (struct fuse_mount *fm,const u64 index) +{ +struct fuse_daxdev_out daxdev_out = {0 }; +struct fuse_conn *fc = fm->fc; +struct famfs_daxdev *daxdev; +int rc; + +FUSE_ARGS (args); + +/* Store the daxdev in our table */ +if (index >= fc->dax_devlist->nslots) { +pr_err ("%s: index(%lld) > nslots(%d)\n" , + __func__, index, fc->dax_devlist->nslots); +return -EINVAL ; + } + + args.opcode =FUSE_GET_DAXDEV ; + args.nodeid = index; + + args.in_numargs =0 ; + + args.out_numargs =1 ; + args.out_args[0 ].size =sizeof (daxdev_out); + args.out_args[0 ].value = &daxdev_out; + +/* Send GET_DAXDEV command */ + rc =fuse_simple_request (fm, &args); +if (rc) { +pr_err ("%s: rc=%d from fuse_simple_request()\n" , + __func__, rc); +/* Error will be that the payload is smaller than FMAP_BUFSIZE, +* which is the max we can handle. Empty payload handled below. +*/ +return rc; + } + + scoped_guard(rwsem_write, &fc->famfs_devlist_sem) { + daxdev = &fc->dax_devlist->devlist[index]; + +/* Abort if daxdev is now valid (races are possible here) */ +if (daxdev->valid) { +pr_debug ("%s: daxdev already known\n" , __func__); +return 0 ; + } + +/* Verify dev is valid and can be opened and gets the devno */ + rc =famfs_verify_daxdev (daxdev_out.name, &daxdev->devno); +if (rc) { +pr_err ("%s: rc=%d from famfs_verify_daxdev()\n" , + __func__, rc); +return rc; + } + + daxdev->name =kstrdup (daxdev_out.name,GFP_KERNEL ); +if (!daxdev->name) +return -ENOMEM ; + +/* This will fail if it's not a dax device */ + daxdev->devp =dax_dev_get (daxdev->devno); +if (!daxdev->devp) { +pr_warn ("%s: device %s not found or not dax\n" , + __func__, daxdev_out.name); +kfree (daxdev->name); + daxdev->name =NULL ; +return -ENODEV ; + } + +wmb ();/* All other fields must be visible before valid */ + daxdev->valid =1 ; + } + +return 0 ; +} + +/** +* famfs_update_daxdev_table() - Update the daxdev table +* @fm: fuse_mount +* @meta: famfs_file_meta, in-memory format, built from a GET_FMAP response +* +* This function is called for each new file fmap, to verify whether all +* referenced daxdevs are already known (i.e. in the table). Any daxdev +* indices referenced in @meta but not in the table will be retrieved via +* famfs_fuse_get_daxdev() and added to the table +* +* Return: 0=success +* -errno=failure +*/ +static int +famfs_update_daxdev_table ( +struct fuse_mount *fm, +const struct famfs_file_meta *meta) +{ +struct famfs_dax_devlist *local_devlist; +struct fuse_conn *fc = fm->fc; +int indices_to_fetch[MAX_DAXDEVS ]; +int n_to_fetch =0 ; +int err; + +/* First time through we will need to allocate the dax_devlist */ +if (!fc->dax_devlist) { + local_devlist =kcalloc (1 ,sizeof (*fc->dax_devlist),GFP_KERNEL ); +if (!local_devlist) +return -ENOMEM ; + + local_devlist->nslots =MAX_DAXDEVS ; + + local_devlist->devlist =kcalloc (MAX_DAXDEVS , +sizeof (struct famfs_daxdev ), +GFP_KERNEL ); +if (!local_devlist->devlist) { +kfree (local_devlist); +return -ENOMEM ; + } + +/* We don't need famfs_devlist_sem here because we use cmpxchg */ +if (cmpxchg (&fc->dax_devlist,NULL , local_devlist) !=NULL ) { +kfree (local_devlist->devlist); +kfree (local_devlist);/* another thread beat us to it */ + } + } + +/* Collect indices that need fetching while holding read lock */ + scoped_guard(rwsem_read, &fc->famfs_devlist_sem) { +unsigned long i; + +for_each_set_bit (i, (unsigned long *)&meta->dev_bitmap,MAX_DAXDEVS ) { +if (!(fc->dax_devlist->devlist[i].valid)) + indices_to_fetch[n_to_fetch++] = i; + } + } + +/* Fetch needed daxdevs outside the read lock */ +for (int j =0 ; j < n_to_fetch; j++) { + err =famfs_fuse_get_daxdev (fm, indices_to_fetch[j]); +if (err) +pr_err ("%s: failed to get daxdev=%d\n" , + __func__, indices_to_fetch[j]); + } + +return 0 ; +}/***************************************************************************/
@@ -184,7 +406,7 @@ famfs_fuse_meta_alloc(/* ie_in = one interleaved extent in fmap_buf */ ie_in = fmap_buf + next_offset; -/* Move past one interleaved extent header in fmap_buf */ +/* Move past 1 interleaved extent header in fmap_buf */ next_offset +=sizeof (*ie_in);if (next_offset > fmap_buf_size) { pr_err("%s:%d: fmap_buf underflow offset/size %ld/%ld\n",
@@ -329,6 +551,9 @@ famfs_file_init_dax(if (rc) goto errout; +/* Make sure this fmap doesn't reference any unknown daxdevs */ +famfs_update_daxdev_table (fm, meta); +/* Publish the famfs metadata on fi->famfs_meta */ inode_lock (inode);
fs/fuse/famfs_kfmap.h +26 -0
@@ -64,4 +64,30 @@ struct famfs_file_meta { }; }; +/* +* famfs_daxdev - tracking struct for a daxdev within a famfs file system +* +* This is the in-memory daxdev metadata that is populated by parsing +* the responses to GET_FMAP messages +*/ +struct famfs_daxdev { +/* Include dev uuid? */ +bool valid; +bool error; +dev_t devno; +struct dax_device *devp; +char *name; +}; + +#define MAX_DAXDEVS 24 + +/* +* famfs_dax_devlist - list of famfs_daxdev's +*/ +struct famfs_dax_devlist { +int nslots; +int ndevs; +struct famfs_daxdev *devlist; +}; +#endif /* FAMFS_KFMAP_H */
fs/fuse/fuse_i.h +19 -0
@@ -1006,6 +1006,11 @@ struct fuse_conn {/* Request timeout (in jiffies). 0 = no timeout */ unsigned int req_timeout; } timeout; + +#if IS_ENABLED (CONFIG_FUSE_FAMFS_DAX ) +struct rw_semaphore famfs_devlist_sem; +struct famfs_dax_devlist *dax_devlist; +#endif };/*
@@ -1647,6 +1652,8 @@ int famfs_file_init_dax(struct fuse_mount *fm, size_t fmap_size);void __famfs_meta_free (void *map); +void famfs_teardown (struct fuse_conn *fc); +/* Set fi->famfs_meta = NULL regardless of prior value */ static inline void famfs_meta_init (struct fuse_inode *fi) {
@@ -1668,6 +1675,11 @@ static inline void famfs_meta_free(struct fuse_inode *fi) } } +static inline void famfs_init_devlist_sem (struct fuse_conn *fc) +{ +init_rwsem (&fc->famfs_devlist_sem); +} +static inline int fuse_file_famfs (struct fuse_inode *fi) {return (READ_ONCE (fi->famfs_meta) !=NULL );
@@ -1677,6 +1689,9 @@ int fuse_get_fmap(struct fuse_mount *fm, struct inode *inode);#else /* !CONFIG_FUSE_FAMFS_DAX */ +static inline void famfs_teardown (struct fuse_conn *fc) +{ +}static inline struct fuse_backing *famfs_meta_set (struct fuse_inode *fi,void *meta) {
@@ -1687,6 +1702,10 @@ static inline void famfs_meta_free(struct fuse_inode *fi) { } +static inline void famfs_init_devlist_sem (struct fuse_conn *fc) +{ +} +static inline int fuse_file_famfs (struct fuse_inode *fi) {return 0 ;
fs/fuse/inode.c +6 -1
@@ -1048,6 +1048,9 @@ void fuse_conn_put(struct fuse_conn *fc)WARN_ON (atomic_read (&bucket->count) !=1 );kfree (bucket); } +if (IS_ENABLED (CONFIG_FUSE_FAMFS_DAX )) +famfs_teardown (fc); +if (IS_ENABLED (CONFIG_FUSE_PASSTHROUGH ))fuse_backing_files_free (fc);call_rcu (&fc->rcu, delayed_release);
@@ -1477,8 +1480,10 @@ static void process_init_reply(struct fuse_mount *fm, struct fuse_args *args,u64 in_flags =FIELD_PREP (GENMASK_ULL (63 ,32 ), ia->in.flags2) | ia->in.flags; -if (in_flags &FUSE_DAX_FMAP ) +if (in_flags &FUSE_DAX_FMAP ) { +famfs_init_devlist_sem (fc); fc->famfs_iomap =1 ; + } } }else { ra_pages = fc->max_read /PAGE_SIZE ;
fs/namei.c +1 -0
@@ -4206,6 +4206,7 @@ bool may_open_dev(const struct path *path)return !(path->mnt->mnt_flags &MNT_NODEV ) && !(path->mnt->mnt_sb->s_iflags &SB_I_NODEV ); } +EXPORT_SYMBOL (may_open_dev);static int may_open (struct mnt_idmap *idmap,const struct path *path,int acc_mode,int flag)
include/uapi/linux/fuse.h +20 -0
@@ -247,6 +247,9 @@* - struct fuse_famfs_simple_ext * - struct fuse_famfs_iext * - struct fuse_famfs_fmap_header +* - Add the following structs for the GET_DAXDEV message and reply +* - struct fuse_get_daxdev_in +* - struct fuse_get_daxdev_out * - Add the following enumerated types * - enum fuse_famfs_file_type * - enum famfs_ext_type
@@ -678,6 +681,7 @@ enum fuse_opcode {/* Famfs / devdax opcodes */ FUSE_GET_FMAP =54 , +FUSE_GET_DAXDEV =55 ,/* CUSE specific operations */ CUSE_INIT = 4096,
@@ -1369,6 +1373,22 @@ struct fuse_famfs_fmap_header {uint64_t reserved1; }; +struct fuse_get_daxdev_in { +uint32_t daxdev_num; +}; + +#define DAXDEV_NAME_MAX 256 + +/* fuse_daxdev_out has enough space for a uuid if we need it */ +struct fuse_daxdev_out { +uint16_t index; +uint16_t reserved; +uint32_t reserved2; +uint64_t reserved3; +uint64_t reserved4; +char name[DAXDEV_NAME_MAX ]; +}; +static inline int32_t fmap_msg_min_size (void ) {/* Smallest fmap message is a header plus one simple extent */
Signature · 1 line
2.53.0
PATCH v8 6/10 famfs_fuse: Plumb dax iomap and fuse read/write/mmap
In reply to John GrovesFrom: John Groves <[email protected]>
This commit fills in read/write/mmap handling for famfs files. The dev_dax_iomap interface is used - just like xfs in fs-dax mode.
- Read/write are handled by famfs_fuse_[read|write]_iter() via dax_iomap_rw() to fsdev_dax.
- Mmap is handled by famfs_fuse_mmap()
- Faults are handled by famfs_filemap_fault(), using dax_iomap_fault() to fsdev_dax.
- File offset to dax offset resolution is handled via famfs_fuse_iomap_begin(), which uses famfs "fmaps" to resolve the the requested (file, offset) to an offset on a dax device (by way of famfs_fileofs_to_daxofs() and famfs_interleave_fileofs_to_daxofs())
--- fs/fuse/famfs.c | 448 +++++++++++++++++++++++++++++++++++++++++++++++ fs/fuse/file.c | 18 +- fs/fuse/fuse_i.h | 19 ++ 3 files changed, 483 insertions(+), 2 deletions(-)
fs/fuse/famfs.c +448 -0
@@ -576,6 +576,454 @@ famfs_file_init_dax(return rc; } +/********************************************************************* +* iomap_operations +* +* This stuff uses the iomap (dax-related) helpers to resolve file offsets to +* offsets within a dax device. +*/ + +static int famfs_file_bad (struct inode *inode); + +static int +famfs_interleave_fileofs_to_daxofs (struct inode *inode,struct iomap *iomap, +loff_t file_offset,off_t len,unsigned int flags) +{ +struct fuse_inode *fi =get_fuse_inode (inode); +struct famfs_file_meta *meta = fi->famfs_meta; +struct fuse_conn *fc =get_fuse_conn (inode); +loff_t local_offset = file_offset; + +/* This function is only for extent_type INTERLEAVED_EXTENT */ +if (meta->fm_extent_type !=INTERLEAVED_EXTENT ) { +pr_err ("%s: bad extent type\n" , __func__); + goto err_out; + } + +if (famfs_file_bad (inode)) + goto err_out; + + iomap->offset = file_offset; + +for (int i =0 ; i < meta->fm_niext; i++) { +struct famfs_meta_interleaved_ext *fei = &meta->ie[i]; +u64 chunk_size = fei->fie_chunk_size; +u64 nstrips = fei->fie_nstrips; +u64 ext_size =min (fei->fie_nbytes, meta->file_size); + +if (!IS_ALIGNED (chunk_size,PMD_SIZE )) { +pr_err ("%s: chunk_size %lld not PMD-aligned\n" , + __func__, meta->ie[i].fie_chunk_size); +return -EINVAL ; + } +if (ext_size ==0 ) { +pr_err ("%s: ext_size=%lld file_size=%ld\n" , + __func__, fei->fie_nbytes, meta->file_size); + goto err_out; + } + +/* Is the data is in this striped extent? */ +if (local_offset < ext_size) { +u64 chunk_num = local_offset / chunk_size; +u64 chunk_offset = local_offset % chunk_size; +u64 chunk_remainder = chunk_size - chunk_offset; +u64 stripe_num = chunk_num / nstrips; +u64 strip_num = chunk_num % nstrips; +u64 strip_offset = chunk_offset + (stripe_num * chunk_size); +u64 strip_dax_ofs = fei->ie_strips[strip_num].ext_offset; +u64 strip_devidx = fei->ie_strips[strip_num].dev_index; + +if (strip_devidx >= fc->dax_devlist->nslots) { +pr_err ("%s: strip_devidx %llu >= nslots %d\n" , + __func__, strip_devidx, + fc->dax_devlist->nslots); + goto err_out; + } + +if (!fc->dax_devlist->devlist[strip_devidx].valid) { +pr_err ("%s: daxdev=%lld invalid\n" , __func__, + strip_devidx); + goto err_out; + } + + iomap->addr = strip_dax_ofs + strip_offset; + iomap->offset = file_offset; + iomap->length =min_t (loff_t, len, chunk_remainder); + + iomap->dax_dev = fc->dax_devlist->devlist[strip_devidx].devp; + + iomap->type =IOMAP_MAPPED ; + iomap->flags = flags; + +return 0 ; + } + local_offset -= ext_size;/* offset is beyond this striped extent */ + } + + err_out: +pr_err ("%s: err_out\n" , __func__); + +/* We fell out the end of the extent list. +* Set iomap to zero length in this case, and return 0 +* This just means that the r/w is past EOF +*/ + iomap->addr =0 ;/* there is no valid dax device offset */ + iomap->offset = file_offset;/* file offset */ + iomap->length =0 ;/* this had better result in no access to dax mem */ + iomap->dax_dev =NULL ; + iomap->type =IOMAP_MAPPED ; + iomap->flags = flags; + +return -EIO ; +} + +/** +* famfs_fileofs_to_daxofs() - Resolve (file, offset, len) to (daxdev, offset, len) +* +* This function is called by famfs_fuse_iomap_begin() to resolve an offset in a +* file to an offset in a dax device. This is upcalled from dax from calls to +* both * dax_iomap_fault() and dax_iomap_rw(). Dax finishes the job resolving +* a fault to a specific physical page (the fault case) or doing a memcpy +* variant (the rw case) +* +* Pages can be PTE (4k), PMD (2MiB) or (theoretically) PuD (1GiB) +* (these sizes are for X86; may vary on other cpu architectures +* +* @inode: The file where the fault occurred +* @iomap: To be filled in to indicate where to find the right memory, +* relative to a dax device. +* @file_offset: Within the file where the fault occurred (will be page boundary) +* @len: The length of the faulted mapping (will be a page multiple) +* (will be trimmed in *iomap if it's disjoint in the extent list) +* @flags: flags passed to famfs_fuse_iomap_begin(), and sent back via +* struct iomap +* +* Return values: 0. (info is returned in a modified @iomap struct) +*/ +static int +famfs_fileofs_to_daxofs (struct inode *inode,struct iomap *iomap, +loff_t file_offset,off_t len,unsigned int flags) +{ +struct fuse_inode *fi =get_fuse_inode (inode); +struct famfs_file_meta *meta = fi->famfs_meta; +struct fuse_conn *fc =get_fuse_conn (inode); +loff_t local_offset = file_offset; + +if (!fc->dax_devlist) { +pr_err ("%s: null dax_devlist\n" , __func__); + goto err_out; + } + +if (famfs_file_bad (inode)) + goto err_out; + +if (meta->fm_extent_type ==INTERLEAVED_EXTENT ) +return famfs_interleave_fileofs_to_daxofs (inode, iomap, + file_offset, + len, flags); + + iomap->offset = file_offset; + +for (int i =0 ; i < meta->fm_nextents; i++) { +/* TODO: check devindex too */ +loff_t dax_ext_offset = meta->se[i].ext_offset; +loff_t dax_ext_len = meta->se[i].ext_len; +u64 daxdev_idx = meta->se[i].dev_index; + + +/* TODO: test that superblock and log offsets only happen +* with superblock and log files. Requires instrumentaiton +* from user space... +*/ + +/* local_offset is the offset minus the size of extents skipped +* so far; If local_offset < dax_ext_len, the data of interest +* starts in this extent +*/ +if (local_offset < dax_ext_len) { +loff_t ext_len_remainder = dax_ext_len - local_offset; +struct famfs_daxdev *dd; + +if (daxdev_idx >= fc->dax_devlist->nslots) { +pr_err ("%s: daxdev_idx %llu >= nslots %d\n" , + __func__, daxdev_idx, + fc->dax_devlist->nslots); + goto err_out; + } + + dd = &fc->dax_devlist->devlist[daxdev_idx]; + +if (!dd->valid || dd->error) { +pr_err ("%s: daxdev=%lld %s\n" , __func__, + daxdev_idx, + dd->valid ?"error" :"invalid" ); + goto err_out; + } + +/* +* OK, we found the file metadata extent where this +* data begins +* @local_offset - The offset within the current +* extent +* @ext_len_remainder - Remaining length of ext after +* skipping local_offset +* Outputs: +* iomap->addr: the offset within the dax device where +* the data starts +* iomap->offset: the file offset +* iomap->length: the valid length resolved here +*/ + iomap->addr = dax_ext_offset + local_offset; + iomap->offset = file_offset; + iomap->length =min_t (loff_t, len, ext_len_remainder); + + iomap->dax_dev = fc->dax_devlist->devlist[daxdev_idx].devp; + + iomap->type =IOMAP_MAPPED ; + iomap->flags = flags; +return 0 ; + } + local_offset -= dax_ext_len;/* Get ready for the next extent */ + } + + err_out: +pr_err ("%s: err_out\n" , __func__); + +/* We fell out the end of the extent list. +* Set iomap to zero length in this case, and return 0 +* This just means that the r/w is past EOF +*/ + iomap->addr =0 ;/* there is no valid dax device offset */ + iomap->offset = file_offset;/* file offset */ + iomap->length =0 ;/* this had better result in no access to dax mem */ + iomap->dax_dev =NULL ; + iomap->type =IOMAP_MAPPED ; + iomap->flags = flags; + +return -EIO ; +} + +/** +* famfs_fuse_iomap_begin() - Handler for iomap_begin upcall from dax +* +* This function is pretty simple because files are +* * never partially allocated +* * never have holes (never sparse) +* * never "allocate on write" +* +* @inode: inode for the file being accessed +* @offset: offset within the file +* @length: Length being accessed at offset +* @flags: flags to be retured via struct iomap +* @iomap: iomap struct to be filled in, resolving (offset, length) to +* (daxdev, offset, len) +* @srcmap: source mapping if it is a COW operation (which it is not here) +*/ +static int +famfs_fuse_iomap_begin (struct inode *inode,loff_t offset,loff_t length, +unsigned int flags,struct iomap *iomap,struct iomap *srcmap) +{ +struct fuse_inode *fi =get_fuse_inode (inode); +struct famfs_file_meta *meta = fi->famfs_meta; +size_t size; + + size =i_size_read (inode); + +WARN_ON (size != meta->file_size); + +return famfs_fileofs_to_daxofs (inode, iomap, offset, length, flags); +} + +/* Note: We never need a special set of write_iomap_ops because famfs never +* performs allocation on write. +*/ +const struct iomap_ops famfs_iomap_ops = { + .iomap_begin = famfs_fuse_iomap_begin, +}; + +/********************************************************************* +* vm_operations +*/ +static vm_fault_t +__famfs_fuse_filemap_fault (struct vm_fault *vmf,unsigned int pe_size, +bool write_fault) +{ +struct inode *inode =file_inode (vmf->vma->vm_file); +vm_fault_t ret; +unsigned long pfn; + +if (!IS_DAX (file_inode (vmf->vma->vm_file))) { +pr_err ("%s: file not marked IS_DAX!!\n" , __func__); +return VM_FAULT_SIGBUS ; + } + +if (write_fault) { +sb_start_pagefault (inode->i_sb); +file_update_time (vmf->vma->vm_file); + } + + ret =dax_iomap_fault (vmf, pe_size, &pfn,NULL , &famfs_iomap_ops); +if (ret &VM_FAULT_NEEDDSYNC ) + ret =dax_finish_sync_fault (vmf, pe_size, pfn); + +if (write_fault) +sb_end_pagefault (inode->i_sb); + +return ret; +} + +static inline bool +famfs_is_write_fault (struct vm_fault *vmf) +{ +return (vmf->flags &FAULT_FLAG_WRITE ) && + (vmf->vma->vm_flags &VM_SHARED ); +} + +static vm_fault_t +famfs_filemap_fault (struct vm_fault *vmf) +{ +return __famfs_fuse_filemap_fault (vmf,0 ,famfs_is_write_fault (vmf)); +} + +static vm_fault_t +famfs_filemap_huge_fault (struct vm_fault *vmf,unsigned int pe_size) +{ +return __famfs_fuse_filemap_fault (vmf, pe_size, +famfs_is_write_fault (vmf)); +} + +static vm_fault_t +famfs_filemap_mkwrite (struct vm_fault *vmf) +{ +return __famfs_fuse_filemap_fault (vmf,0 , true); +} + +const struct vm_operations_struct famfs_file_vm_ops = { + .fault = famfs_filemap_fault, + .huge_fault = famfs_filemap_huge_fault, + .map_pages = filemap_map_pages, + .page_mkwrite = famfs_filemap_mkwrite, + .pfn_mkwrite = famfs_filemap_mkwrite, +}; + +/********************************************************************* +* file_operations +*/ + +/** +* famfs_file_bad() - Check for files that aren't in a valid state +* +* @inode: inode +* +* Returns: 0=success +* -errno=failure +*/ +static int +famfs_file_bad (struct inode *inode) +{ +struct fuse_inode *fi =get_fuse_inode (inode); +struct famfs_file_meta *meta = fi->famfs_meta; +size_t i_size =i_size_read (inode); + +if (!meta) { +pr_err ("%s: un-initialized famfs file\n" , __func__); +return -EIO ; + } +if (meta->error) { +pr_debug ("%s: previously detected metadata errors\n" , __func__); +return -EIO ; + } +if (i_size != meta->file_size) { +pr_warn ("%s: i_size overwritten from %ld to %ld\n" , + __func__, meta->file_size, i_size); + meta->error = true; +return -ENXIO ; + } +if (!IS_DAX (inode)) { +pr_debug ("%s: inode %llx IS_DAX is false\n" , + __func__, (u64 )inode); +return -ENXIO ; + } +return 0 ; +} + +static ssize_t +famfs_fuse_rw_prep (struct kiocb *iocb,struct iov_iter *ubuf) +{ +struct inode *inode = iocb->ki_filp->f_mapping->host; +size_t i_size =i_size_read (inode); +size_t count =iov_iter_count (ubuf); +size_t max_count; +ssize_t rc; + + rc =famfs_file_bad (inode); +if (rc) +return (ssize_t )rc; + +/* Avoid unsigned underflow if position is past EOF */ +if (iocb->ki_pos >= i_size) + max_count =0 ; +else + max_count = i_size - iocb->ki_pos; + +if (count > max_count) +iov_iter_truncate (ubuf, max_count); + +if (!iov_iter_count (ubuf)) +return 0 ; + +return rc; +} + +ssize_t +famfs_fuse_read_iter (struct kiocb *iocb,struct iov_iter *to) +{ +ssize_t rc; + + rc =famfs_fuse_rw_prep (iocb, to); +if (rc) +return rc; + +if (!iov_iter_count (to)) +return 0 ; + + rc =dax_iomap_rw (iocb, to, &famfs_iomap_ops); + +file_accessed (iocb->ki_filp); +return rc; +} + +ssize_t +famfs_fuse_write_iter (struct kiocb *iocb,struct iov_iter *from) +{ +ssize_t rc; + + rc =famfs_fuse_rw_prep (iocb, from); +if (rc) +return rc; + +if (!iov_iter_count (from)) +return 0 ; + +return dax_iomap_rw (iocb, from, &famfs_iomap_ops); +} + +int +famfs_fuse_mmap (struct file *file,struct vm_area_struct *vma) +{ +struct inode *inode =file_inode (file); +ssize_t rc; + + rc =famfs_file_bad (inode); +if (rc) +return rc; + +file_accessed (file); + vma->vm_ops = &famfs_file_vm_ops; +vm_flags_set (vma,VM_HUGEPAGE ); +return 0 ; +} +#define FMAP_BUFSIZE PAGE_SIZEint fuse_get_fmap (struct fuse_mount *fm,struct inode *inode)
fs/fuse/file.c +16 -2
@@ -1831,6 +1831,8 @@ static ssize_t fuse_file_read_iter(struct kiocb *iocb, struct iov_iter *to)if (FUSE_IS_VIRTIO_DAX (fi))return fuse_dax_read_iter (iocb, to); +if (fuse_file_famfs (fi)) +return famfs_fuse_read_iter (iocb, to);/* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */ if (ff->open_flags &FOPEN_DIRECT_IO )
@@ -1853,6 +1855,8 @@ static ssize_t fuse_file_write_iter(struct kiocb *iocb, struct iov_iter *from)if (FUSE_IS_VIRTIO_DAX (fi))return fuse_dax_write_iter (iocb, from); +if (fuse_file_famfs (fi)) +return famfs_fuse_write_iter (iocb, from);/* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */ if (ff->open_flags &FOPEN_DIRECT_IO )
@@ -1868,9 +1872,13 @@ static ssize_t fuse_splice_read(struct file *in, loff_t *ppos, unsigned int flags) {struct fuse_file *ff = in->private_data; +struct inode *inode =file_inode (in); +struct fuse_inode *fi =get_fuse_inode (inode);/* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */ -if (fuse_file_passthrough (ff) && !(ff->open_flags &FOPEN_DIRECT_IO )) +if (fuse_file_famfs (fi)) +return -EIO ;/* famfs does not use the page cache... */ +else if (fuse_file_passthrough (ff) && !(ff->open_flags &FOPEN_DIRECT_IO ))return fuse_passthrough_splice_read (in, ppos, pipe, len, flags);else return filemap_splice_read (in, ppos, pipe, len, flags);
@@ -1880,9 +1888,13 @@ static ssize_t fuse_splice_write(struct pipe_inode_info *pipe, struct file *out, loff_t *ppos, size_t len, unsigned int flags) { struct fuse_file *ff = out->private_data; +struct inode *inode =file_inode (out); +struct fuse_inode *fi =get_fuse_inode (inode);/* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */ -if (fuse_file_passthrough (ff) && !(ff->open_flags &FOPEN_DIRECT_IO )) +if (fuse_file_famfs (fi)) +return -EIO ;/* famfs does not use the page cache... */ +else if (fuse_file_passthrough (ff) && !(ff->open_flags &FOPEN_DIRECT_IO ))return fuse_passthrough_splice_write (pipe, out, ppos, len, flags);else return iter_file_splice_write (pipe, out, ppos, len, flags);
@@ -2390,6 +2402,8 @@ static int fuse_file_mmap(struct file *file, struct vm_area_struct *vma)/* DAX mmap is superior to direct_io mmap */ if (FUSE_IS_VIRTIO_DAX (fi))return fuse_dax_mmap (file, vma); +if (fuse_file_famfs (fi)) +return famfs_fuse_mmap (file, vma);/* * If inode is in passthrough io mode, because it has some file open
fs/fuse/fuse_i.h +19 -0
@@ -1650,6 +1650,9 @@ extern void fuse_sysctl_unregister(void);int famfs_file_init_dax (struct fuse_mount *fm,struct inode *inode,void *fmap_buf,size_t fmap_size); +ssize_t famfs_fuse_write_iter (struct kiocb *iocb,struct iov_iter *from); +ssize_t famfs_fuse_read_iter (struct kiocb *iocb,struct iov_iter *to); +int famfs_fuse_mmap (struct file *file,struct vm_area_struct *vma);void __famfs_meta_free (void *map);void famfs_teardown (struct fuse_conn *fc);
@@ -1692,6 +1695,22 @@ int fuse_get_fmap(struct fuse_mount *fm, struct inode *inode);static inline void famfs_teardown (struct fuse_conn *fc) { } +static inline ssize_t famfs_fuse_write_iter (struct kiocb *iocb, +struct iov_iter *to) +{ +return -ENODEV ; +} +static inline ssize_t famfs_fuse_read_iter (struct kiocb *iocb, +struct iov_iter *to) +{ +return -ENODEV ; +} +static inline int famfs_fuse_mmap (struct file *file, +struct vm_area_struct *vma) +{ +return -ENODEV ; +} +static inline struct fuse_backing *famfs_meta_set (struct fuse_inode *fi,void *meta) {
Signature · 1 line
2.53.0
PATCH v8 7/10 famfs_fuse: Add holder_operations for dax notify_failure()
In reply to John GrovesFrom: John Groves <[email protected]>
Memory errors are at least somewhat more likely on disaggregated memory than on-board memory. This commit registers to be notified by fsdev_dax in the event that a memory failure is detected.
When a file access resolves to a daxdev with memory errors, it will fail with an appropriate error.
If a daxdev failed fs_dax_get(), we set dd->dax_err. If a daxdev called our notify_failure(), set dd->error. When any of the above happens, set (file)->error and stop allowing access.
In general, the recovery from memory errors is to unmount the file system and re-initialize the memory, but there may be usable degraded modes of operation - particularly in the future when famfs supports file systems backed by more than one daxdev. In those cases, accessing data that is on a working daxdev can still work.
For now, return errors for any file that has encountered a memory or dax error.
--- fs/fuse/famfs.c | 110 +++++++++++++++++++++++++++++++++++++++--- fs/fuse/famfs_kfmap.h | 3 +- 2 files changed, 105 insertions(+), 8 deletions(-)
fs/fuse/famfs.c +103 -7
@@ -21,6 +21,26 @@#include "famfs_kfmap.h" #include "fuse_i.h" +static void famfs_set_daxdev_err ( +struct fuse_conn *fc,struct dax_device *dax_devp); + +static int +famfs_dax_notify_failure (struct dax_device *dax_devp,u64 offset, +u64 len,int mf_flags) +{ +struct fuse_conn *fc =dax_holder (dax_devp); + +famfs_set_daxdev_err (fc, dax_devp); + +return 0 ; +} + +static const struct dax_holder_operations famfs_fuse_dax_holder_ops = { + .notify_failure = famfs_dax_notify_failure, +}; + +/*****************************************************************************/ +/* * famfs_teardown() *
@@ -47,9 +67,12 @@ famfs_teardown(struct fuse_conn *fc)if (!dd->valid)continue ; -/* Release reference from dax_dev_get() */ -if (dd->devp) +/* Only call fs_put_dax if fs_dax_get succeeded */ +if (dd->devp) { +if (!dd->dax_err) +fs_put_dax (dd->devp, fc);put_dax (dd->devp); + }kfree (dd->name); }
@@ -169,6 +192,17 @@ famfs_fuse_get_daxdev(struct fuse_mount *fm, const u64 index)return -ENODEV ; } + rc =fs_dax_get (daxdev->devp, fc, &famfs_fuse_dax_holder_ops); +if (rc) { +/* Mark as valid with dax_err to prevent retry loop. +* famfs_dax_err() will return -EIO on access attempts. +* Teardown handles this case: skips fs_put_dax, calls put_dax. +*/ + daxdev->dax_err =1 ; +pr_err ("%s: fs_dax_get(%lld) failed\n" , + __func__, (u64 )daxdev->devno); + } +wmb ();/* All other fields must be visible before valid */ daxdev->valid =1 ; }
@@ -244,6 +278,36 @@ famfs_update_daxdev_table(return 0 ; } +static void +famfs_set_daxdev_err ( +struct fuse_conn *fc, +struct dax_device *dax_devp) +{ +int i; + +/* Gotta search the list by dax_devp; +* read lock because we're not adding or removing daxdev entries +*/ + scoped_guard(rwsem_write, &fc->famfs_devlist_sem) { +for (i =0 ; i < fc->dax_devlist->nslots; i++) { +if (fc->dax_devlist->devlist[i].valid) { +struct famfs_daxdev *dd; + + dd = &fc->dax_devlist->devlist[i]; +if (dd->devp != dax_devp) +continue ; + + dd->error = true; + +pr_err ("%s: memory error on daxdev %s (%d)\n" , + __func__, dd->name, i); +return ; + } + } + } +pr_err ("%s: memory err on unrecognized daxdev\n" , __func__); +} +/***************************************************************************/ void __famfs_meta_free (void *famfs_meta)
@@ -585,6 +649,26 @@ famfs_file_init_dax(static int famfs_file_bad (struct inode *inode); +static int famfs_dax_err (struct famfs_daxdev *dd) +{ +if (!dd->valid) { +pr_err ("%s: daxdev=%s invalid\n" , + __func__, dd->name); +return -EIO ; + } +if (dd->dax_err) { +pr_err ("%s: daxdev=%s dax_err\n" , + __func__, dd->name); +return -EIO ; + } +if (dd->error) { +pr_err ("%s: daxdev=%s memory error\n" , + __func__, dd->name); +return -EHWPOISON ; + } +return 0 ; +} +static int famfs_interleave_fileofs_to_daxofs (struct inode *inode,struct iomap *iomap,loff_t file_offset,off_t len,unsigned int flags)
@@ -624,6 +708,7 @@ famfs_interleave_fileofs_to_daxofs(struct inode *inode, struct iomap *iomap,/* Is the data is in this striped extent? */ if (local_offset < ext_size) { +struct famfs_daxdev *dd;u64 chunk_num = local_offset / chunk_size;u64 chunk_offset = local_offset % chunk_size;u64 chunk_remainder = chunk_size - chunk_offset;
@@ -632,6 +717,7 @@ famfs_interleave_fileofs_to_daxofs(struct inode *inode, struct iomap *iomap,u64 strip_offset = chunk_offset + (stripe_num * chunk_size);u64 strip_dax_ofs = fei->ie_strips[strip_num].ext_offset;u64 strip_devidx = fei->ie_strips[strip_num].dev_index; +int rc;if (strip_devidx >= fc->dax_devlist->nslots) { pr_err("%s: strip_devidx %llu >= nslots %d\n",
@@ -646,6 +732,15 @@ famfs_interleave_fileofs_to_daxofs(struct inode *inode, struct iomap *iomap, goto err_out; } + dd = &fc->dax_devlist->devlist[strip_devidx]; + + rc =famfs_dax_err (dd); +if (rc) { +/* Shut down access to this file */ + meta->error = true; +return rc; + } + iomap->addr = strip_dax_ofs + strip_offset; iomap->offset = file_offset; iomap->length =min_t (loff_t, len, chunk_remainder);
@@ -743,6 +838,7 @@ famfs_fileofs_to_daxofs(struct inode *inode, struct iomap *iomap,if (local_offset < dax_ext_len) {loff_t ext_len_remainder = dax_ext_len - local_offset;struct famfs_daxdev *dd; +int rc;if (daxdev_idx >= fc->dax_devlist->nslots) { pr_err("%s: daxdev_idx %llu >= nslots %d\n",
@@ -753,11 +849,11 @@ famfs_fileofs_to_daxofs(struct inode *inode, struct iomap *iomap, dd = &fc->dax_devlist->devlist[daxdev_idx]; -if (!dd->valid || dd->error) { -pr_err ("%s: daxdev=%lld %s\n" , __func__, - daxdev_idx, - dd->valid ?"error" :"invalid" ); - goto err_out; + rc =famfs_dax_err (dd); +if (rc) { +/* Shut down access to this file */ + meta->error = true; +return rc; }/*
fs/fuse/famfs_kfmap.h +2 -1
@@ -73,7 +73,8 @@ struct famfs_file_meta {struct famfs_daxdev {/* Include dev uuid? */ bool valid; -bool error; +bool error;/* Dax has reported a memory error (probably poison) */ +bool dax_err;/* fs_dax_get() failed */ dev_t devno;struct dax_device *devp;char *name;
Signature · 1 line
2.53.0
PATCH v8 8/10 famfs_fuse: Add DAX address_space_operations with noop_dirty_folio
In reply to John GrovesFrom: John Groves <[email protected]>
Famfs is memory-backed; there is no place to write back to, and no reason to mark pages dirty at all.
--- fs/fuse/famfs.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
fs/fuse/famfs.c +11 -0
@@ -14,6 +14,7 @@#include <linux/mm.h> #include <linux/dax.h> #include <linux/iomap.h> +#include <linux/pagemap.h> #include <linux/path.h> #include <linux/namei.h> #include <linux/string.h>
@@ -39,6 +40,15 @@ static const struct dax_holder_operations famfs_fuse_dax_holder_ops = { .notify_failure = famfs_dax_notify_failure, }; +/* +* DAX address_space_operations for famfs. +* famfs doesn't need dirty tracking - writes go directly to +* memory with no writeback required. +*/ +static const struct address_space_operations famfs_dax_aops = { + .dirty_folio = noop_dirty_folio, +}; +/*****************************************************************************/ /*
@@ -624,6 +634,7 @@ famfs_file_init_dax(if (famfs_meta_set (fi, meta) ==NULL ) {i_size_write (inode, meta->file_size); inode->i_flags |=S_DAX ; + inode->i_data.a_ops = &famfs_dax_aops; }else {pr_debug ("%s: file already had metadata\n" , __func__);__famfs_meta_free (meta);
Signature · 1 line
2.53.0
PATCH v8 9/10 famfs_fuse: Add famfs fmap metadata documentation
In reply to John GrovesFrom: John Groves <[email protected]>
This describes the fmap metadata - both simple and interleaved
--- fs/fuse/famfs_kfmap.h | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+)
fs/fuse/famfs_kfmap.h +73 -0
@@ -7,6 +7,79 @@#ifndef FAMFS_KFMAP_H #define FAMFS_KFMAP_H +/* KABI version 43 (aka v2) fmap structures +* +* The location of the memory backing for a famfs file is described by +* the response to the GET_FMAP fuse message (defined in +* include/uapi/linux/fuse.h +* +* There are currently two extent formats: Simple and Interleaved. +* +* Simple extents are just (devindex, offset, length) tuples, where devindex +* references a devdax device that must be retrievable via the GET_DAXDEV +* message/response. +* +* The extent list size must be >= file_size. +* +* Interleaved extents merit some additional explanation. Interleaved +* extents stripe data across a collection of strips. Each strip is a +* contiguous allocation from a single devdax device - and is described by +* a simple_extent structure. +* +* Interleaved_extent example: +* ie_nstrips = 4 +* ie_chunk_size = 2MiB +* ie_nbytes = 24MiB +* +* ┌────────────┐────────────┐────────────┐────────────┐ +* │Chunk = 0 │Chunk = 1 │Chunk = 2 │Chunk = 3 │ +* │Strip = 0 │Strip = 1 │Strip = 2 │Strip = 3 │ +* │Stripe = 0 │Stripe = 0 │Stripe = 0 │Stripe = 0 │ +* │ │ │ │ │ +* └────────────┘────────────┘────────────┘────────────┘ +* │Chunk = 4 │Chunk = 5 │Chunk = 6 │Chunk = 7 │ +* │Strip = 0 │Strip = 1 │Strip = 2 │Strip = 3 │ +* │Stripe = 1 │Stripe = 1 │Stripe = 1 │Stripe = 1 │ +* │ │ │ │ │ +* └────────────┘────────────┘────────────┘────────────┘ +* │Chunk = 8 │Chunk = 9 │Chunk = 10 │Chunk = 11 │ +* │Strip = 0 │Strip = 1 │Strip = 2 │Strip = 3 │ +* │Stripe = 2 │Stripe = 2 │Stripe = 2 │Stripe = 2 │ +* │ │ │ │ │ +* └────────────┘────────────┘────────────┘────────────┘ +* +* * Data is laid out across chunks in chunk # order +* * Columns are strips +* * Strips are contiguous devdax extents, normally each coming from a +* different memory device +* * Rows are stripes +* * The number of chunks is (int)((file_size + chunk_size - 1) / chunk_size) +* (and obviously the last chunk could be partial) +* * The stripe_size = (nstrips * chunk_size) +* * chunk_num(offset) = offset / chunk_size //integer division +* * strip_num(offset) = chunk_num(offset) % nchunks +* * stripe_num(offset) = offset / stripe_size //integer division +* * ...You get the idea - see the code for more details... +* +* Some concrete examples from the layout above: +* * Offset 0 in the file is offset 0 in chunk 0, which is offset 0 in +* strip 0 +* * Offset 4MiB in the file is offset 0 in chunk 2, which is offset 0 in +* strip 2 +* * Offset 15MiB in the file is offset 1MiB in chunk 7, which is offset +* 3MiB in strip 3 +* +* Notes about this metadata format: +* +* * For various reasons, chunk_size must be a multiple of the applicable +* PAGE_SIZE +* * Since chunk_size and nstrips are constant within an interleaved_extent, +* resolving a file offset to a strip offset within a single +* interleaved_ext is order 1. +* * If nstrips==1, a list of interleaved_ext structures degenerates to a +* regular extent list (albeit with some wasted struct space). +*/ +/* * The structures below are the in-memory metadata format for famfs files. * Metadata retrieved via the GET_FMAP response is converted to this format
Signature · 1 line
2.53.0
PATCH v8 10/10 famfs_fuse: Add documentation
In reply to John GrovesFrom: John Groves <[email protected]>
Add Documentation/filesystems/famfs.rst and update MAINTAINERS
--- Documentation/filesystems/famfs.rst | 142 ++++++++++++++++++++++++++++ Documentation/filesystems/index.rst | 1 + MAINTAINERS | 1 + 3 files changed, 144 insertions(+) create mode 100644 Documentation/filesystems/famfs.rst
Documentation/filesystems/famfs.rst new file +142 -0
@@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _famfs_index: + +================================================================== +famfs: The fabric-attached memory file system +================================================================== + +- Copyright (C) 2024-2026 Micron Technology, Inc. + +Introduction +============ +Compute Express Link (CXL) provides a mechanism for disaggregated or +fabric-attached memory (FAM). This creates opportunities for data sharing; +clustered apps that would otherwise have to shard or replicate data can +share one copy in disaggregated memory. + +Famfs, which is not CXL-specific in any way, provides a mechanism for +multiple hosts to concurrently access data in shared memory, by giving it +a file system interface. With famfs, any app that understands files can +access data sets in shared memory. Although famfs supports read and write, +the real point is to support mmap, which provides direct (dax) access to +the memory - either writable or read-only. + +Shared memory can pose complex coherency and synchronization issues, but +there are also simple cases. Two simple and eminently useful patterns that +occur frequently in data analytics and AI are: + +* Serial Sharing - Only one host or process at a time has access to a file +* Read-only Sharing - Multiple hosts or processes share read-only access + to a file + +The famfs fuse file system is part of the famfs framework; user space +components [1] handle metadata allocation and distribution, and provide a +low-level fuse server to expose files that map directly to [presumably +shared] memory. + +The famfs framework manages coherency of its own metadata and structures, +but does not attempt to manage coherency for applications. + +Famfs also provides data isolation between files. That is, even though +the host has access to an entire memory "device" (as a devdax device), apps +cannot write to memory for which the file is read-only, and mapping one +file provides isolation from the memory of all other files. This is pretty +basic, but some experimental shared memory usage patterns provide no such +isolation. + +Principles of Operation +======================= + +Famfs is a file system with one or more devdax devices as a first-class +backing device(s). Metadata maintenance and query operations happen +entirely in user space. + +The famfs low-level fuse server daemon provides file maps (fmaps) and +devdax device info to the fuse/famfs kernel component so that +read/write/mapping faults can be handled without up-calls for all active +files. + +The famfs user space is responsible for maintaining and distributing +consistent metadata. This is currently handled via an append-only +metadata log within the memory, but this is orthogonal to the fuse/famfs +kernel code. + +Once instantiated, "the same file" on each host points to the same shared +memory, but in-memory metadata (inodes, etc.) is ephemeral on each host +that has a famfs instance mounted. Use cases are free to allow or not +allow mutations to data on a file-by-file basis. + +When an app accesses a data object in a famfs file, there is no page cache +involvement. The CPU cache is loaded directly from the shared memory. In +some use cases, this is an enormous reduction in read amplification +compared to loading an entire page into the page cache. + + +Famfs is Not a Conventional File System +--------------------------------------- + +Famfs files can be accessed by conventional means, but there are +limitations. The kernel component of fuse/famfs is not involved in the +allocation of backing memory for files at all; the famfs user space +creates files and responds as a low-level fuse server with fmaps and +devdax device info upon request. + +Famfs differs in some important ways from conventional file systems: + +* Files must be pre-allocated by the famfs framework; allocation is never + performed on (or after) write. +* Any operation that changes a file's size is considered to put the file + in an invalid state, disabling access to the data. It may be possible to + revisit this in the future. (Typically the famfs user space can restore + files to a valid state by replaying the famfs metadata log.) + +Famfs exists to apply the existing file system abstractions to shared +memory so applications and workflows can more easily adapt to an +environment with disaggregated shared memory. + +Memory Error Handling +===================== + +Possible memory errors include timeouts, poison, and unexpected +reconfiguration of an underlying dax device. In all of these cases, famfs +receives a call from the devdax layer via its iomap_ops->notify_failure() +function. If any memory errors have been detected, access to the affected +daxdev is disabled to avoid further errors or corruption. + +In all known cases, famfs can be unmounted cleanly. In most cases errors +can be cleared by re-initializing the memory - at which point a new famfs +file system can be created. + +Key Requirements +================ + +The primary requirements for famfs are: + +1. Must support a file system abstraction backed by sharable devdax memory +2. Files must efficiently handle VMA faults +3. Must support metadata distribution in a sharable way +4. Must handle clients with a stale copy of metadata + +The famfs kernel component takes care of 1-2 above by caching each file's +mapping metadata in the kernel. + +Requirements 3 and 4 are handled by the user space components, and are +largely orthogonal to the functionality of the famfs kernel module. + +Requirements 3 and 4 cannot be met by conventional fs-dax file systems +(e.g. xfs) because they use write-back metadata; it is not valid to mount +such a file system on two hosts from the same in-memory image. + + +Famfs Usage +=========== + +Famfs usage is documented at [1]. + + +References +========== + +- [1] Famfs user space repository and documentation + https://github.com/cxl-micron-reskit/famfs
Documentation/filesystems/index.rst +1 -0
@@ -89,6 +89,7 @@ Documentation for filesystem implementations. ext3 ext4/index f2fs + famfs gfs2/index hfs hfsplus
MAINTAINERS +1 -0
@@ -10516,6 +10516,7 @@ M: John Groves <[email protected]> L: [email protected] L: [email protected] S: Supported +F: Documentation/filesystems/famfs.rst F: fs/fuse/famfs.c F: fs/fuse/famfs_kfmap.h
Signature · 1 line
2.53.0
kernel test robot
build botRe: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table
In reply to John Groveskernel test robot noticed the following build errors:
Hi John,
kernel test robot noticed the following build errors:
[auto build test ERROR on mszeredi-fuse/for-next]
[also build test ERROR on brauner-vfs/vfs.all linus/master v7.0-rc4 next-20260319] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/John-Groves/famfs_fuse-Basic-fuse-kernel-ABI-enablement-for-famfs/20260320-152719 base: https://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git for-next patch link: https://lore.kernel.org/r/0100019d06402f14-3a614309-878f-4837-be72-e36fd75d74f6-000000%40email.amazonses.com patch subject: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table config: x86_64-kexec (https://download.01.org/0day-ci/archive/20260320/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260320/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> fs/fuse/famfs.c:163:18: error: call to undeclared function 'dax_dev_get'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
163 | daxdev->devp = dax_dev_get(daxdev->devno);
| ^
>> fs/fuse/famfs.c:163:16: error: incompatible integer to pointer conversion assigning to 'struct dax_device *' from 'int' [-Wint-conversion]
163 | daxdev->devp = dax_dev_get(daxdev->devno);
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~2 errors generated.
vim +/dax_dev_get +163 fs/fuse/famfs.c
90
91 /**
92 * famfs_fuse_get_daxdev() - Retrieve info for a DAX device from fuse server
93 *
94 * Send a GET_DAXDEV message to the fuse server to retrieve info on a
95 * dax device.
96 *
97 * @fm: fuse_mount
98 * @index: the index of the dax device; daxdevs are referred to by index
99 * in fmaps, and the server resolves the index to a particular daxdev
100 *
101 * Returns: 0=success
102 * -errno=failure
103 */
104 static int
105 famfs_fuse_get_daxdev(struct fuse_mount *fm, const u64 index)
106 {
107 struct fuse_daxdev_out daxdev_out = { 0 };
108 struct fuse_conn *fc = fm->fc;
109 struct famfs_daxdev *daxdev;
110 int rc;
111
112 FUSE_ARGS(args);
113
114 /* Store the daxdev in our table */
115 if (index >= fc->dax_devlist->nslots) {
116 pr_err("%s: index(%lld) > nslots(%d)\n",
117 __func__, index, fc->dax_devlist->nslots);
118 return -EINVAL;
119 }
120
121 args.opcode = FUSE_GET_DAXDEV;
122 args.nodeid = index;
123
124 args.in_numargs = 0;
125
126 args.out_numargs = 1;
127 args.out_args[0].size = sizeof(daxdev_out);
128 args.out_args[0].value = &daxdev_out;
129
130 /* Send GET_DAXDEV command */
131 rc = fuse_simple_request(fm, &args);
132 if (rc) {
133 pr_err("%s: rc=%d from fuse_simple_request()\n",
134 __func__, rc);
135 /* Error will be that the payload is smaller than FMAP_BUFSIZE,
136 * which is the max we can handle. Empty payload handled below.
137 */
138 return rc;
139 }
140
141 scoped_guard(rwsem_write, &fc->famfs_devlist_sem) {
142 daxdev = &fc->dax_devlist->devlist[index];
143
144 /* Abort if daxdev is now valid (races are possible here) */
145 if (daxdev->valid) {
146 pr_debug("%s: daxdev already known\n", __func__);
147 return 0;
148 }
149
150 /* Verify dev is valid and can be opened and gets the devno */
151 rc = famfs_verify_daxdev(daxdev_out.name, &daxdev->devno);
152 if (rc) {
153 pr_err("%s: rc=%d from famfs_verify_daxdev()\n",
154 __func__, rc);
155 return rc;
156 }
157
158 daxdev->name = kstrdup(daxdev_out.name, GFP_KERNEL);
159 if (!daxdev->name)
160 return -ENOMEM;
161
162 /* This will fail if it's not a dax device */
> 163 daxdev->devp = dax_dev_get(daxdev->devno); 164 if (!daxdev->devp) {
165 pr_warn("%s: device %s not found or not dax\n",
166 __func__, daxdev_out.name);
167 kfree(daxdev->name);
168 daxdev->name = NULL;
169 return -ENODEV;
170 }
171
172 wmb(); /* All other fields must be visible before valid */
173 daxdev->valid = 1;
174 }
175
176 return 0;
177 }
178
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table
In reply to John Groveskernel test robot noticed the following build errors:
Hi John,
kernel test robot noticed the following build errors:
[auto build test ERROR on mszeredi-fuse/for-next]
[also build test ERROR on brauner-vfs/vfs.all linus/master v7.0-rc4 next-20260320] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/John-Groves/famfs_fuse-Basic-fuse-kernel-ABI-enablement-for-famfs/20260320-152719 base: https://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git for-next patch link: https://lore.kernel.org/r/0100019d06402f14-3a614309-878f-4837-be72-e36fd75d74f6-000000%40email.amazonses.com patch subject: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20260321/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260321/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
fs/fuse/famfs.c: In function 'famfs_fuse_get_daxdev':
>> fs/fuse/famfs.c:163:32: error: implicit declaration of function 'dax_dev_get' [-Wimplicit-function-declaration]
163 | daxdev->devp = dax_dev_get(daxdev->devno);
| ^~~~~~~~~~~
>> fs/fuse/famfs.c:163:30: error: assignment to 'struct dax_device *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
163 | daxdev->devp = dax_dev_get(daxdev->devno);
| ^vim +/dax_dev_get +163 fs/fuse/famfs.c
90
91 /**
92 * famfs_fuse_get_daxdev() - Retrieve info for a DAX device from fuse server
93 *
94 * Send a GET_DAXDEV message to the fuse server to retrieve info on a
95 * dax device.
96 *
97 * @fm: fuse_mount
98 * @index: the index of the dax device; daxdevs are referred to by index
99 * in fmaps, and the server resolves the index to a particular daxdev
100 *
101 * Returns: 0=success
102 * -errno=failure
103 */
104 static int
105 famfs_fuse_get_daxdev(struct fuse_mount *fm, const u64 index)
106 {
107 struct fuse_daxdev_out daxdev_out = { 0 };
108 struct fuse_conn *fc = fm->fc;
109 struct famfs_daxdev *daxdev;
110 int rc;
111
112 FUSE_ARGS(args);
113
114 /* Store the daxdev in our table */
115 if (index >= fc->dax_devlist->nslots) {
116 pr_err("%s: index(%lld) > nslots(%d)\n",
117 __func__, index, fc->dax_devlist->nslots);
118 return -EINVAL;
119 }
120
121 args.opcode = FUSE_GET_DAXDEV;
122 args.nodeid = index;
123
124 args.in_numargs = 0;
125
126 args.out_numargs = 1;
127 args.out_args[0].size = sizeof(daxdev_out);
128 args.out_args[0].value = &daxdev_out;
129
130 /* Send GET_DAXDEV command */
131 rc = fuse_simple_request(fm, &args);
132 if (rc) {
133 pr_err("%s: rc=%d from fuse_simple_request()\n",
134 __func__, rc);
135 /* Error will be that the payload is smaller than FMAP_BUFSIZE,
136 * which is the max we can handle. Empty payload handled below.
137 */
138 return rc;
139 }
140
141 scoped_guard(rwsem_write, &fc->famfs_devlist_sem) {
142 daxdev = &fc->dax_devlist->devlist[index];
143
144 /* Abort if daxdev is now valid (races are possible here) */
145 if (daxdev->valid) {
146 pr_debug("%s: daxdev already known\n", __func__);
147 return 0;
148 }
149
150 /* Verify dev is valid and can be opened and gets the devno */
151 rc = famfs_verify_daxdev(daxdev_out.name, &daxdev->devno);
152 if (rc) {
153 pr_err("%s: rc=%d from famfs_verify_daxdev()\n",
154 __func__, rc);
155 return rc;
156 }
157
158 daxdev->name = kstrdup(daxdev_out.name, GFP_KERNEL);
159 if (!daxdev->name)
160 return -ENOMEM;
161
162 /* This will fail if it's not a dax device */
> 163 daxdev->devp = dax_dev_get(daxdev->devno); 164 if (!daxdev->devp) {
165 pr_warn("%s: device %s not found or not dax\n",
166 __func__, daxdev_out.name);
167 kfree(daxdev->name);
168 daxdev->name = NULL;
169 return -ENODEV;
170 }
171
172 wmb(); /* All other fields must be visible before valid */
173 daxdev->valid = 1;
174 }
175
176 return 0;
177 }
178
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
John Groves
maintainerRe: [PATCH V8 2/8] dax: Factor out dax_folio_reset_order() helper
In reply to Jonathan CameronQuoted 21 lines · Jonathan Cameron
Quoted 18 lines · On Wed, 18 Mar 2026 20:28:20 -0500 John Groves <[email protected]> wrote:
From: John Groves <[email protected]>
Both fs/dax.c:dax_folio_put() and
drivers/dax/fsdev.c:fsdev_clear_folio_state()(the latter coming in the next commit after this one) contain nearly identical code to reset a compound DAX folio back to order-0 pages. Factor this out into a shared helper function.The new
dax_folio_reset_order()function:
- Clears the folio's mapping and share count
- Resets compound folio state viafolio_reset_order()
- Clears PageHead andcompound_headfor each sub-page
- Restores the pgmap pointer for each resulting order-0 folio
- Returns the original folio order (for callers that need to advance by that many pages)This simplifies
fsdev_clear_folio_state()from ~50 lines to ~15 lines while maintaining the same functionality in both call sites.
Quoted 7 lines
Quoted 4 lines
Suggested-by Jonathan Cameron <[email protected]>Reviewed-by Ira Weiny <[email protected]>Reviewed-by Dave Jiang <[email protected]>Signed-off-by John Groves <[email protected]>Comment below. I may well be needing more coffee, or failing wrt to background knowledge as I only occasionally dip into dax.
thanks!
Quoted 60 lines
Quoted 21 lines
--- fs/dax.c | 60 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 42 insertions(+), 18 deletions(-)fs/dax.c +7
-0@@ -378,6 +378,45 @@ static void dax_folio_make_shared(struct folio *folio) folio->share =1 ; } +/** +* dax_folio_reset_order - Reset a compound DAX folio to order-0 pages +* @folio: The folio to reset +* +* Splits a compound folio back into individual order-0 pages, +* clearing compound state and restoring pgmap pointers. +* fs/dax.c static void dax_folio_make_shared(struct folio *folio) · line 397
Quoted 10 lines
fs/dax.c +10
-0+* Returns: the original folio order (0 if already order-0) +*/ +int dax_folio_reset_order (struct folio *folio) +{ +struct dev_pagemap *pgmap =page_pgmap (&folio->page); +int order =folio_order (folio); +int i; + + folio->mapping =NULL ; + folio->share =0 ;This is different from the code you are replacing..
Just above the call to this in
dax_folio_put()if (!dax_folio_is_shared(folio))
// in here is the interesting bit...ref = 0; else //this is fine because either it's still > 0 and we return //or it is zero and you are writing that again. ref = --folio->share; if (ref) return ref;So the path that bothers me is if !dax_folio_is_shared() can return false with shared != 0
/*
* A DAX folio is considered shared if it has no mapping set and ->share (which
* shares the ->index field) is non-zero. Note this may return false even if the
* page is shared between multiple files but has not yet actually been mapped
* into multiple address spaces.
*/
static inline booldax_folio_is_shared(struct folio *folio){ return !folio->mapping && folio->share; }
Quoted 8 lines
So it can if !folio->mapping is false (i.e.
folio->mappingis set)Now I have zero idea of whether this is a real path and have
a long review queue so not looking into it for now.
However if it's not then I'd expect some commentary in the patch description to say why it's not a problem. Maybe even a precursor patch adding thefolio->shareso there is a place to state clearly that it doesn't matter and why.
I believe it is correct, and I'm adding a clarifying comment above as follows:
/* * DAX maintains the invariant that folio->share != 0 only when * folio->mapping == NULL (enforced by dax_folio_make_shared()). * Equivalently: folio->mapping != NULL implies folio->share == 0. * Callers ensure share has been decremented to zero before calling * here, so unconditionally clearing both fields is correct. */ folio->mapping = NULL; folio->share = 0; ...
Quoted 5 lines
fs/dax.c static void dax_folio_make_shared(struct folio *folio) · line 400
Quoted 4 lines
fs/dax.c +3
-0+ +if (!order) { + folio->pgmap = pgmap;This is also different...
Here too, I think it is correct, and I'm adding a comment as follows:
if (!order) {
/*
* Restore pgmap explicitly even for order-0 folios. For the
* dax_folio_put() caller this is a no-op (same value), but
* fsdev_clear_folio_state() may call this on folios that were
* previously compound and need pgmap re-established.
*/
folio->pgmap = pgmap;
return 0;
}...but if I'm missing anything I hope somebody will point it out!
Quoted 11 lines
fs/dax.c static void dax_folio_make_shared(struct folio *folio) · line 406
Quoted 7 lines
fs/dax.c +6
-0+return 0 ; + } + +folio_reset_order (folio); + +for (i =0 ; i < (1UL << order); i++) {I'd take advantage of evolving conventions and do
for (int i = 0; i < ...)
Done, thanks!
John
<snip>
John Groves
maintainerRe: [PATCH V8 3/8] dax: add fsdev.c driver for fs-dax on character dax
In reply to Jonathan CameronQuoted 71 lines · Jonathan Cameron
Quoted 19 lines · On Wed, 18 Mar 2026 20:28:37 -0500 John Groves <[email protected]> wrote:
The new fsdev driver provides pages/folios initialized compatibly with fsdax - normal rather than devdax-style refcounting, and starting out with order-0 folios.
When fsdev binds to a daxdev, it is usually (always?) switching from the devdax mode (
device.c), which pre-initializes compound folios according to its alignment. Fsdev usesfsdev_clear_folio_state()to switch the folios into a fsdax-compatible state.A side effect of this is that raw mmap doesn't (can't?) work on an fsdev dax instance. Accordingly, The fsdev driver does not provide raw mmap - devices must be put in 'devdax' mode (
drivers/dax/device.c) to get raw mmap capability.In this commit is just the framework, which remaps pages/folios compatibly with fsdax.
Enabling dax changes:
Quoted 7 lines
-
bus.h: addDAXDRV_FSDEV_TYPEdriver type
-bus.c: allowDAXDRV_FSDEV_TYPEdrivers to bind to daxdevs
-dax.h: prototypeinode_dax(), which fsdev needsSuggested-by Dan Williams <[email protected]>Suggested-by Gregory Price <[email protected]>Signed-off-by John Groves <[email protected]>A few comments inline. I think some of the code here could be moved to a helper library used by both this and
device.cQuoted 38 lines
--- MAINTAINERS | 8 ++ drivers/dax/Makefile | 6 + drivers/dax/bus.c | 4 + drivers/dax/bus.h | 1 + drivers/dax/fsdev.c | 253 +++++++++++++++++++++++++++++++++++++++++++ fs/dax.c | 1 + include/linux/dax.h | 3 + 7 files changed, 276 insertions(+) create mode 100644 drivers/dax/fsdev.cMAINTAINERS +8
-0@@ -7298,6 +7298,14 @@ L: [email protected] S: Supported F: drivers/dax/ +DEVICE DIRECT ACCESS (DAX) [fsdev_dax] +M: John Groves <[email protected]> +M: John Groves <[email protected]> +L: [email protected] +L: [email protected] +S: Supported +F: drivers/dax/fsdev.c + DEVICE FREQUENCY (DEVFREQ) M: MyungJoo Ham <[email protected]> M: Kyungmin Park <[email protected]>drivers/dax/Makefile +0
-0@@ -5,10 +5,16 @@ obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o obj-$(CONFIG_DEV_DAX_CXL) += dax_cxl.o
Quoted 7 lines
drivers/dax/Makefile obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o · line 11
Quoted 4 lines
drivers/dax/Makefile +4
-0+# fsdev_dax: fs-dax compatible devdax driver (needs DEV_DAX and FS_DAX) +ifeq ($(CONFIG_FS_DAX),y) +obj-$(CONFIG_DEV_DAX) += fsdev_dax.o +endifWhy not throw in a new
CONFIG_FSDAX_DEVand handle the dependencies inKconfig?
At one point I had another config parameter, but I'm trying not to gratuitously add them. The fsdev driver is pretty small, and including it whenever FS_DAX is enabled felt reasonable to me. I'm willing to change it if there's a consensus that way.
Quoted 17 lines
Quoted 16 lines
drivers/dax/Makefile +2
-0+ dax-y := super.o dax-y += bus.o device_dax-y := device.o dax_pmem-y := pmem.o dax_cxl-y := cxl.o +fsdev_dax-y := fsdev.o obj-y += hmem/drivers/dax/fsdev.c new file +0
-0
Quoted 8 lines
drivers/dax/fsdev.c line 124
Quoted 6 lines
drivers/dax/fsdev.c new file +6
-0+static int fsdev_dax_probe (struct dev_dax *dev_dax) +{ +struct dax_device *dax_dev = dev_dax->dax_dev; +struct device *dev = &dev_dax->dev; +struct dev_pagemap *pgmap; +u64 data_offset =0 ;See below. I think you can useful reduce scope of this one.
As of now, I've reduced the scope, but in the very next commit it needs to move back here. So meh...not sure that's worth it for one commit
Quoted 9 lines
drivers/dax/fsdev.c line 129
Quoted 6 lines
drivers/dax/fsdev.c new file +5
-0+struct inode *inode; +struct cdev *cdev; +void *addr; +int rc, i; +There is a lot of duplication in here with
dax/device.cIs any of it suitable for shared helpers?
I haven't addressed factoring out more duplicated code yet. Ideally I'd like to do that after the initial merge, but I'm paying attention to whether there's pressure to do it.
Quoted 12 lines
Quoted 12 lines
drivers/dax/fsdev.c new file +11
-0+if (static_dev_dax (dev_dax)) { +if (dev_dax->nr_range >1 ) { +dev_warn (dev,"static pgmap / multi-range device conflict\n" ); +return -EINVAL ; + } + + pgmap = dev_dax->pgmap; + }else { +size_t pgmap_size; + +if (dev_dax->pgmap) {
Quoted 8 lines
drivers/dax/fsdev.c line 146
Quoted 6 lines
drivers/dax/fsdev.c new file +6
-0+dev_warn (dev,"dynamic-dax with pre-populated page map\n" ); +return -EINVAL ; + } + + pgmap_size =struct_size (pgmap, ranges, dev_dax->nr_range -1 ); + pgmap =devm_kzalloc (dev, pgmap_size,GFP_KERNEL );Bonus space before
GFP_KERNEL.
Excised, thanks
Quoted 20 lines
Quoted 20 lines
drivers/dax/fsdev.c new file +18
-0+if (!pgmap) +return -ENOMEM ; + + pgmap->nr_range = dev_dax->nr_range; + dev_dax->pgmap = pgmap; + +for (i =0 ; i < dev_dax->nr_range; i++) { +struct range *range = &dev_dax->ranges[i].range; + + pgmap->ranges[i] = *range; + } + } + +for (i =0 ; i < dev_dax->nr_range; i++) { +struct range *range = &dev_dax->ranges[i].range; + +if (!devm_request_mem_region (dev, range->start, +range_len (range),dev_name (dev))) {
Quoted 8 lines
drivers/dax/fsdev.c line 169
Quoted 5 lines
drivers/dax/fsdev.c new file +5
-0+dev_warn (dev,"mapping%d: %#llx-%#llx could not reserve range\n" , + i, range->start, range->end); +return -EBUSY ; + } + }Everything above here is shared. Some sort of
_init()or similar library function seems in order.
Taken under advisement. Will look at this soon.
Quoted 40 lines
Quoted 40 lines
drivers/dax/fsdev.c new file +39
-0+ +/* +* FS-DAX compatible mode: Use MEMORY_DEVICE_FS_DAX type and +* do NOT set vmemmap_shift. This leaves folios at order-0, +* allowing fs-dax to dynamically create compound folios as needed +* (similar to pmem behavior). +*/ + pgmap->type =MEMORY_DEVICE_FS_DAX ; + pgmap->ops = &fsdev_pagemap_ops; + pgmap->owner = dev_dax; + +/* +* CRITICAL DIFFERENCE from device.c: +* We do NOT set vmemmap_shift here, even if align > PAGE_SIZE. +* This ensures folios remain order-0 and are compatible with +* fs-dax's folio management. +*/ + + addr =devm_memremap_pages (dev, pgmap); +if (IS_ERR (addr)) +return PTR_ERR (addr); + +/* +* Clear any stale compound folio state left over from a previous +* driver (e.g., device_dax with vmemmap_shift). Also register this +* as a devm action so folio state is cleared on unbind, ensuring +* clean pages for subsequent drivers (e.g., kmem for system-ram). +*/ +fsdev_clear_folio_state (dev_dax); + rc =devm_add_action_or_reset (dev, fsdev_clear_folio_state_action, + dev_dax); +if (rc) +return rc; + +/* Detect whether the data is at a non-zero offset into the memory */ +if (pgmap->range.start != dev_dax->ranges[0 ].range.start) { +u64 phys = dev_dax->ranges[0 ].range.start; +u64 pgmap_phys = dev_dax->pgmap[0 ].range.start; +
Quoted 8 lines
drivers/dax/fsdev.c line 213
Quoted 5 lines
drivers/dax/fsdev.c new file +5
-0+if (!WARN_ON (pgmap_phys > phys)) + data_offset = phys - pgmap_phys; + +pr_debug ("%s: offset detected phys=%llx pgmap_phys=%llx offset=%llx\n" , + __func__, phys, pgmap_phys, data_offset);Might change later, but at least at this point you could pull declaration of
data_offsetinto this scope.
done as of now, but it's used right after the closing brace of this block in the very next commit.
Quoted 26 lines
Quoted 26 lines
drivers/dax/fsdev.c new file +18
-0+ } + + inode =dax_inode (dax_dev); + cdev = inode->i_cdev; +cdev_init (cdev, &fsdev_fops); + cdev->owner = dev->driver->owner; +cdev_set_parent (cdev, &dev->kobj); + rc =cdev_add (cdev, dev->devt,1 ); +if (rc) +return rc; + + rc =devm_add_action_or_reset (dev, fsdev_cdev_del, cdev); +if (rc) +return rc; + +run_dax (dax_dev); +return devm_add_action_or_reset (dev, fsdev_kill, dev_dax); +}include/linux/dax.h +0
-0@@ -51,6 +51,7 @@ struct dax_holder_operations {
Quoted 5 lines
include/linux/dax.h struct dax_holder_operations { · line 54
Quoted 3 lines
include/linux/dax.h +1
-0#if IS_ENABLED (CONFIG_DAX )struct dax_device *alloc_dax (void *private,const struct dax_operations *ops); +Unrelated change. Tidy this up for v9.
Spurious blank line dropped - thanks
Quoted 8 lines
Quoted 8 lines
include/linux/dax.h +0
-0void *dax_holder (struct dax_device *dax_dev);void put_dax (struct dax_device *dax_dev);void kill_dax (struct dax_device *dax_dev);@@ -151,8 +152,10 @@ static inline void fs_put_dax(struct dax_device *dax_dev, void *holder)#endif /* CONFIG_BLOCK && CONFIG_FS_DAX */
Quoted 4 lines
include/linux/dax.h static inline void fs_put_dax(struct dax_device *dax_dev, void *holder) · line 155
Quoted 2 lines
include/linux/dax.h +1
-0#if IS_ENABLED (CONFIG_FS_DAX ) +struct dax_device *inode_dax (struct inode *inode);Already in
dax_private.hso why does it want to be here?
Indeed, thanks!
Regards,
John
John Groves
maintainerRe: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table
In reply to kernel test robotQuoted 25 lines · kernel test robot
Hi John,
kernel test robot noticed the following build errors:
[auto build test ERROR on mszeredi-fuse/for-next]
[also build test ERROR on brauner-vfs/vfs.all linus/master v7.0-rc4 next-20260320] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/John-Groves/famfs_fuse-Basic-fuse-kernel-ABI-enablement-for-famfs/20260320-152719 base: https://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git for-next patch link: https://lore.kernel.org/r/0100019d06402f14-3a614309-878f-4837-be72-e36fd75d74f6-000000%40email.amazonses.com patch subject: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20260321/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260321/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 7 lines
fs/fuse/famfs.c: In function 'famfs_fuse_get_daxdev':Quoted 1 line
Quoted 1 line
fs/fuse/famfs.c:163:32: error: implicit declaration of function 'dax_dev_get' [-Wimplicit-function-declaration]163 | daxdev->devp = dax_dev_get(daxdev->devno); | ^~~~~~~~~~~Quoted 1 line
Quoted 1 line
fs/fuse/famfs.c:163:30: error: assignment to 'structdax_device*' from 'int' makes pointer from integer without a cast [-Wint-conversion]163 | daxdev->devp = dax_dev_get(daxdev->devno); | ^
The fuse_famfs patches are dependent on the famfs dax patches (which include the missing dax_dev_get() function. The plan that we have discussed from the dax side is for Ira to pull in the famfs dax patches, and then for Miklos to pull that branch from Ira, and base the fuse_famfs on top of that.
I'll be pushing some minor revisions to the dax series early this week. At that point I hope that series is ready.
Looks like the famfs series is no longer in Miklos' for-next; does this approach work for you, Miklos?
Thanks,
John
Re: [PATCH V8 3/8] dax: add fsdev.c driver for fs-dax on character dax
In reply to John GrovesQuoted 17 lines
Quoted 9 lines
Quoted 9 lines
drivers/dax/Makefile +0
-0@@ -5,10 +5,16 @@ obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o obj-$(CONFIG_DEV_DAX_CXL) += dax_cxl.oQuoted 7 lines
drivers/dax/Makefile obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o · line 11
Quoted 4 lines
drivers/dax/Makefile +4
-0+# fsdev_dax: fs-dax compatible devdax driver (needs DEV_DAX and FS_DAX) +ifeq ($(CONFIG_FS_DAX),y) +obj-$(CONFIG_DEV_DAX) += fsdev_dax.o +endifWhy not throw in a new
CONFIG_FSDAX_DEVand handle the dependencies inKconfig?
Quoted 4 lines
At one point I had another config parameter, but I'm trying not to gratuitously add them. The fsdev driver is pretty small, and including it whenever
FS_DAXis enabled felt reasonable to me. I'm willing to change it if there's a consensus that way.
You can make the build do exactly the same thing with a separate Kconfig option. Just moves where the dependency tracking is. I'd prefer Kconfig because that's generally where I'd look for something like this.
Jonathan
John Groves
maintainerRe: [PATCH V8 3/8] dax: add fsdev.c driver for fs-dax on character dax
Quoted 22 lines · Jonathan Cameron
Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
drivers/dax/Makefile +0
-0@@ -5,10 +5,16 @@ obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o obj-$(CONFIG_DEV_DAX_PMEM) += dax_pmem.o obj-$(CONFIG_DEV_DAX_CXL) += dax_cxl.oQuoted 7 lines
drivers/dax/Makefile obj-$(CONFIG_DEV_DAX_KMEM) += kmem.o · line 11
Quoted 4 lines
drivers/dax/Makefile +4
-0+# fsdev_dax: fs-dax compatible devdax driver (needs DEV_DAX and FS_DAX) +ifeq ($(CONFIG_FS_DAX),y) +obj-$(CONFIG_DEV_DAX) += fsdev_dax.o +endifWhy not throw in a new
CONFIG_FSDAX_DEVand handle the dependencies inKconfig?Quoted 4 lines
At one point I had another config parameter, but I'm trying not to gratuitously add them. The fsdev driver is pretty small, and including it whenever
FS_DAXis enabled felt reasonable to me. I'm willing to change it if there's a consensus that way.
Quoted 6 lines
You can make the build do exactly the same thing with a separate
Kconfigoption. Just moves where the dependency tracking is. I'd preferKconfigbecause that's generally where I'd look for something like this.Jonathan
OK, will do. It will be CONFIG_DEV_DAX_FSDEV for naming consistency.
V9 coming within 24 hours...
John
[email protected]
AI reviewRe: [PATCH V8 04/10] famfs_fuse: Create files with famfs fmaps
In reply to John GrovesThank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider: - [High] Integer overflow and undefined behavior when calculating `dev_bitmap` with a 32-bit signed shift.…
Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [High] Integer overflow and undefined behavior when calculating dev_bitmap with a 32-bit signed shift.
- [High] Recursive inode_lock() deadlock on O_TRUNC file open.
- [Medium] Data truncation of file size and mapping size on 32-bit architectures.
- [Medium] Dynamic toggling of S_DAX flag introduces a data race and violates VFS caching invariants.
- [Low] Dangling pointer window in famfs_meta_free due to incorrect cleanup ordering.
- [Low] Meaningless local cmpxchg race check acts as dead code.
- [Low] Uninitialized variable accessed and printed in error message.
- [Low] Implicit cast of untrusted UAPI enum to internal enum.
--
--- Patch [4]: [PATCH V8 04/10] famfs_fuse: Create files with famfs fmaps --- commit 588d2cb1a397d88983186c80e9fec874ef42a12e
Author: John Groves <[email protected]>
famfs_fuse: Create files with famfs fmaps
This patch introduces the ability to allocate and initialize in-memory famfs file metadata mapped to DAX memory. It adds structures for resolving file maps and sets up the initialization path based on FUSE GET_FMAP responses.
Quoted 2 lines
fs/fuse/famfs.c +0
-0
[ ... ]
Quoted 7 lines
fs/fuse/famfs.c +7
-0+static int +famfs_fuse_meta_alloc ( +void *fmap_buf, +size_t fmap_buf_size, +struct famfs_file_meta **metap) +{ +struct fuse_famfs_fmap_header *fmh;
[ ... ]
fs/fuse/famfs.c line 121
Quoted 3 lines
fs/fuse/famfs.c +3
-0+ meta->error = false; + meta->file_type = fmh->file_type; + meta->file_size = fmh->file_size;
[Severity: Medium]
Does this truncate the file size on 32-bit architectures?
The FUSE protocol correctly provides this as a 64-bit value, but the famfs_file_meta structure declares file_size as a size_t. On 32-bit systems, size_t is a 32-bit unsigned integer.
Any file size larger than 4GB will be silently truncated when assigned here, which will later result in incorrect sizes passed to i_size_write().
fs/fuse/famfs.c line 122
Quoted 1 line
fs/fuse/famfs.c +1
-0+ meta->fm_extent_type = fmh->ext_type;
[Severity: Low]
Is it safe to directly assign this FUSE UAPI enum to the internal kernel enum?
The code assigns a UAPI uint32_t (enum famfs_ext_type) directly to a kernel internal enum famfs_extent_type. While the integer values currently align identically, this bypasses type safety and will fail silently if the enumerated types ever diverge.
Quoted 4 lines
fs/fuse/famfs.c +4
-0+ +switch (fmh->ext_type) { +case FUSE_FAMFS_EXT_SIMPLE : { +struct fuse_famfs_simple_ext *se_in;
[ ... ]
fs/fuse/famfs.c line 155
Quoted 7 lines
fs/fuse/famfs.c +7
-0+for (i =0 ; i < fmh->nextents; i++) { + meta->se[i].dev_index = se_in[i].se_devindex; + meta->se[i].ext_offset = se_in[i].se_offset; + meta->se[i].ext_len = se_in[i].se_len; + +/* Record bitmap of referenced daxdev indices */ + meta->dev_bitmap |= (1 << meta->se[i].dev_index);
[Severity: High]
Can this shift operation invoke undefined behavior or corrupt the bitmap?
The FUSE server provides a dev_index which is used to set a bit in a 64-bit bitmap. Because 1 is a signed 32-bit integer literal, a dev_index of 31 results in a negative value that sign-extends to 0xFFFFFFFF80000000 when promoted to u64, incorrectly setting bits 32-63.
If dev_index is 32 or greater, the shift invokes undefined behavior. Using 1ULL would prevent this.
Quoted 9 lines
fs/fuse/famfs.c +9
-0+ + errs +=famfs_check_ext_alignment (&meta->se[i]); + + extent_total += meta->se[i].ext_len; + } +break ; + } + +case FUSE_FAMFS_EXT_INTERLEAVE : {
[ ... ]
Quoted 9 lines
fs/fuse/famfs.c +9
-0+for (i =0 ; i < niext; i++) { +u64 nstrips; +struct fuse_famfs_simple_ext *sie_in; + +/* ie_in = one interleaved extent in fmap_buf */ + ie_in = fmap_buf + next_offset; + +/* Move past one interleaved extent header in fmap_buf */ + next_offset +=sizeof (*ie_in);
fs/fuse/famfs.c line 198
Quoted 10 lines
fs/fuse/famfs.c +10
-0+if (next_offset > fmap_buf_size) { +pr_err ("%s:%d: fmap_buf underflow offset/size %ld/%ld\n" , + __func__, __LINE__, next_offset, + fmap_buf_size); +return -EINVAL ; + } + +if (!IS_ALIGNED (ie_in->ie_chunk_size,PMD_SIZE )) { +pr_err ("%s: chunk_size %lld not PMD-aligned\n" , + __func__, meta->ie[i].fie_chunk_size);
[Severity: Low]
Does this error message print an uninitialized value?
During the interleaved extent alignment checks, if ie_chunk_size is not PMD-aligned, the error message attempts to log meta->ie[i].fie_chunk_size.
However, this field has not yet been assigned from the FUSE payload. The assignment happens several lines below this check, so this will print 0 instead of the actual invalid chunk size.
Quoted 2 lines
fs/fuse/famfs.c +2
-0+return -EINVAL ; + }
[ ... ]
Quoted 4 lines
fs/fuse/famfs.c +4
-0+/* Inner loop is over strips */ +for (j =0 ; j < nstrips; j++) { +struct famfs_meta_simple_ext *strips_out; +u64 devindex = sie_in[j].se_devindex;
fs/fuse/famfs.c line 253
Quoted 10 lines
fs/fuse/famfs.c +10
-0+u64 offset = sie_in[j].se_offset; +u64 len = sie_in[j].se_len; + + strips_out = meta->ie[i].ie_strips; + strips_out[j].dev_index = devindex; + strips_out[j].ext_offset = offset; + strips_out[j].ext_len = len; + +/* Record bitmap of referenced daxdev indices */ + meta->dev_bitmap |= (1 << devindex);
[Severity: High]
As above, can this shift operation invoke undefined behavior or corrupt the bitmap if devindex is 32 or greater?
Quoted 6 lines
fs/fuse/famfs.c +6
-0+ + extent_total += len; + errs +=famfs_check_ext_alignment (&strips_out[j]); + size_remainder -= len; + } + }
[ ... ]
fs/fuse/famfs.c line 290
Quoted 4 lines
fs/fuse/famfs.c +4
-0+if (cmpxchg (metap,NULL , meta) !=NULL ) { +pr_debug ("%s: fmap race detected\n" , __func__); +return 0 ;/* fmap already installed */ + }
[Severity: Low]
Does this cmpxchg check accomplish its intended race detection?
The metap argument points to a purely local stack variable inside the caller famfs_file_init_dax(). Since this variable is completely local to the thread, the cmpxchg will always succeed.
This makes the race detection dead code. The actual functional race check correctly occurs later via famfs_meta_set().
Quoted 7 lines
fs/fuse/famfs.c +7
-0+retain_and_null_ptr (meta); + +return 0 ; +} + +/** +* famfs_file_init_dax() - init famfs dax file metadata
[ ... ]
Quoted 10 lines
fs/fuse/famfs.c +10
-0+int +famfs_file_init_dax ( +struct fuse_mount *fm, +struct inode *inode, +void *fmap_buf, +size_t fmap_size) +{ +struct fuse_inode *fi =get_fuse_inode (inode); +struct famfs_file_meta *meta =NULL ; +int rc;
[ ... ]
fs/fuse/famfs.c line 333
Quoted 6 lines
fs/fuse/famfs.c +6
-0+ rc =famfs_fuse_meta_alloc (fmap_buf, fmap_size, &meta); +if (rc) + goto errout; + +/* Publish the famfs metadata on fi->famfs_meta */ +inode_lock (inode);
[Severity: High]
Could this unconditional lock acquisition cause an A-A self-deadlock?
When a file is opened with O_TRUNC and writeback caching is enabled, the fuse_open() path acquires the inode_lock to perform truncation.
If the filesystem is mounted with famfs iomap support, the open path calls fuse_get_fmap(), which in turn calls famfs_file_init_dax().
This function unconditionally calls inode_lock() again to install the metadata. Since the same thread already holds the lock, it will deadlock.
fs/fuse/famfs.c line 337
Quoted 4 lines
fs/fuse/famfs.c +4
-0+ +if (famfs_meta_set (fi, meta) ==NULL ) { +i_size_write (inode, meta->file_size); + inode->i_flags |=S_DAX ;
[Severity: Medium]
Is it safe to dynamically toggle the DAX flag on an active inode?
This modifies the flags using a read-modify-write without the standard inode_set_flags() helper.
More importantly, changing S_DAX after an inode has been instantiated breaks VFS invariants. If page cache pages were populated prior to this flag change, subsequent I/O will use DAX paths while stale page cache pages remain, potentially triggering VM_BUG_ON or silent data corruption.
Quoted 8 lines
fs/fuse/famfs.c +8
-0+ }else { +pr_debug ("%s: file already had metadata\n" , __func__); +__famfs_meta_free (meta); +/* rc is 0 - the file is valid */ + } + +inode_unlock (inode); +return 0 ;
[ ... ]
Quoted 3 lines
fs/fuse/famfs_kfmap.h new file +0
-0@@ -0,0 +1,67 @@
[ ... ]
fs/fuse/famfs_kfmap.h line 52
Quoted 7 lines
fs/fuse/famfs_kfmap.h new file +7
-0+/* +* Each famfs dax file has this hanging from its fuse_inode->famfs_meta +*/ +struct famfs_file_meta { +bool error; +enum famfs_file_type file_type; +size_t file_size;
[Severity: Medium]
Should this be a u64 or loff_t?
As mentioned earlier, declaring file_size as a size_t on 32-bit systems limits it to a 32-bit unsigned integer, causing truncation for files larger than 4GB.
Quoted 2 lines
fs/fuse/famfs_kfmap.h new file +2
-0+enum famfs_extent_type fm_extent_type; +u64 dev_bitmap;/* bitmap of referenced daxdevs by index */
[ ... ]
Quoted 6 lines
fs/fuse/fuse_i.h +0
-0@@ -1642,6 +1642,9 @@ extern void fuse_sysctl_unregister(void);/* famfs.c */ #if IS_ENABLED (CONFIG_FUSE_FAMFS_DAX )
[ ... ]
fs/fuse/fuse_i.h static inline struct fuse_backing *famfs_meta_set(struct fuse_inode *fi, · line 1669
Quoted 8 lines
static inline void
famfs_meta_free(struct fuse_inode *fi){fs/fuse/fuse_i.h +4
-1-famfs_meta_set (fi,NULL ); +if (fi->famfs_meta !=NULL ) { +__famfs_meta_free (fi->famfs_meta); +famfs_meta_set (fi,NULL ); + } }
[Severity: Low]
Does this create a dangling pointer window?
The metadata memory is freed via __famfs_meta_free() before the pointer is nullified by famfs_meta_set().
While this occurs during inode eviction where concurrency is limited, it creates a brief use-after-free window and breaks the expected safety guarantees of a cleanup helper. Reversing the order would be safer.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table
In reply to John GrovesThank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider: - [High] The FUSE server-provided string daxdev_out.name is not guaranteed to be null-terminated, causing…
Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [High] The FUSE server-provided string daxdev_out.name is not guaranteed to be null-terminated, causing out-of-bounds reads.
- [High] An untrusted device index from the FUSE payload is used in a bitwise shift without prior bounds checking, causing Undefined Behavior.
- [High] The return value of famfs_update_daxdev_table() is ignored, allowing file initialization to proceed with uninitialized DAX devices.
- [Medium] FUSE_GET_DAXDEV ignores the defined input payload struct and incorrectly uses the nodeid field.
- [Medium] Casting u64* to unsigned long* for for_each_set_bit() breaks on 32-bit big-endian systems.
- [Medium] Unprotected plain read of fc->dax_devlist creates a data race with its atomic initialization.
--
--- Patch [5]: [PATCH V8 05/10] famfs_fuse: GET_DAXDEV message and daxdev_table --- commit 5e7c03f2362ee3cfd1a9cae3fbaaaeb19bced93d
Author: John Groves <[email protected]>
famfs_fuse: GET_DAXDEV message and daxdev_table
The new GET_DAXDEV message/response is added. The famfs_teardown() function is added as a primary teardown function for famfs. The command is triggered by the update_daxdev_table() call, if there are any daxdevs in the subject fmap that are not represented in the daxdev_table yet.
Quoted 5 lines
fs/fuse/famfs.c +0
-0@@ -21,6 +21,228 @@
[ ... ]
Quoted 11 lines
fs/fuse/famfs.c +11
-0+static int +famfs_fuse_get_daxdev (struct fuse_mount *fm,const u64 index) +{ +struct fuse_daxdev_out daxdev_out = {0 }; +struct fuse_conn *fc = fm->fc; +struct famfs_daxdev *daxdev; +int rc; + +FUSE_ARGS (args); + +/* Store the daxdev in our table */
fs/fuse/famfs.c line 124
Quoted 10 lines
fs/fuse/famfs.c +10
-0+if (index >= fc->dax_devlist->nslots) { +pr_err ("%s: index(%lld) > nslots(%d)\n" , + __func__, index, fc->dax_devlist->nslots); +return -EINVAL ; + } + + args.opcode =FUSE_GET_DAXDEV ; + args.nodeid = index; + + args.in_numargs =0 ;
[Severity: Medium]
The UAPI defines struct fuse_get_daxdev_in with daxdev_num for the input payload. Does this implementation intentionally ignore that struct, set in_numargs = 0, and repurpose the nodeid field (which is normally for VFS inode numbers) to pass the device index instead? This appears to break the protocol format.
Quoted 8 lines
fs/fuse/famfs.c +8
-0+ + args.out_numargs =1 ; + args.out_args[0 ].size =sizeof (daxdev_out); + args.out_args[0 ].value = &daxdev_out; + +/* Send GET_DAXDEV command */ + rc =fuse_simple_request (fm, &args); +if (rc) {
[ ... ]
Quoted 4 lines
fs/fuse/famfs.c +4
-0+return rc; + } + + scoped_guard(rwsem_write, &fc->famfs_devlist_sem) {
fs/fuse/famfs.c line 151
Quoted 10 lines
fs/fuse/famfs.c +10
-0+ daxdev = &fc->dax_devlist->devlist[index]; + +/* Abort if daxdev is now valid (races are possible here) */ +if (daxdev->valid) { +pr_debug ("%s: daxdev already known\n" , __func__); +return 0 ; + } + +/* Verify dev is valid and can be opened and gets the devno */ + rc =famfs_verify_daxdev (daxdev_out.name, &daxdev->devno);
[Severity: High]
Is the FUSE server-provided string daxdev_out.name guaranteed to be null-terminated?
If a FUSE server sends a response where the 256-byte name field contains no null bytes, passing it to kern_path() (via famfs_verify_daxdev()) and kstrdup() could read past the end of the stack-allocated daxdev_out struct until a null byte is found, potentially causing an out-of-bounds memory read.
[ ... ]
fs/fuse/famfs.c line 204
Quoted 13 lines
fs/fuse/famfs.c +13
-0+static int +famfs_update_daxdev_table ( +struct fuse_mount *fm, +const struct famfs_file_meta *meta) +{ +struct famfs_dax_devlist *local_devlist; +struct fuse_conn *fc = fm->fc; +int indices_to_fetch[MAX_DAXDEVS ]; +int n_to_fetch =0 ; +int err; + +/* First time through we will need to allocate the dax_devlist */ +if (!fc->dax_devlist) {
[Severity: Medium]
Could this unprotected plain read of fc->dax_devlist race with the atomic cmpxchg() publication later in the function? Should this use READ_ONCE() to prevent compiler optimization issues or KCSAN warnings?
Quoted 16 lines
fs/fuse/famfs.c +16
-0+ local_devlist =kcalloc (1 ,sizeof (*fc->dax_devlist),GFP_KERNEL ); +if (!local_devlist) +return -ENOMEM ; + + local_devlist->nslots =MAX_DAXDEVS ; + + local_devlist->devlist =kcalloc (MAX_DAXDEVS , +sizeof (struct famfs_daxdev ), +GFP_KERNEL ); +if (!local_devlist->devlist) { +kfree (local_devlist); +return -ENOMEM ; + } + +/* We don't need famfs_devlist_sem here because we use cmpxchg */ +if (cmpxchg (&fc->dax_devlist,NULL , local_devlist) !=NULL ) {
fs/fuse/famfs.c line 230
Quoted 10 lines
fs/fuse/famfs.c +10
-0+kfree (local_devlist->devlist); +kfree (local_devlist);/* another thread beat us to it */ + } + } + +/* Collect indices that need fetching while holding read lock */ + scoped_guard(rwsem_read, &fc->famfs_devlist_sem) { +unsigned long i; + +for_each_set_bit (i, (unsigned long *)&meta->dev_bitmap,MAX_DAXDEVS ) {
[Severity: Medium]
Does casting a u64 * to unsigned long * here break on 32-bit big-endian systems?
On those architectures, the loop might read the upper 32 bits (which are zero) instead of the lower 32 bits, resulting in a failure to fetch any DAX devices. Should meta->dev_bitmap use DECLARE_BITMAP to avoid endianness and size mismatch bugs?
[ ... ]
fs/fuse/famfs.c famfs_file_init_dax( · line 556
Quoted 7 lines
fs/fuse/famfs.c +3
-0@@ -329,6 +551,9 @@ famfs_file_init_dax(if (rc) goto errout; +/* Make sure this fmap doesn't reference any unknown daxdevs */ +famfs_update_daxdev_table (fm, meta); +
[Severity: High]
The return value of famfs_update_daxdev_table() is an error code, but it is completely ignored here.
If it fails (e.g., due to an ENOMEM allocation failure or if the daxdev fetch fails), file initialization proceeds with uninitialized DAX devices. Could this lead to invalid state and potential NULL pointer dereferences on subsequent accesses?
fs/fuse/famfs.c famfs_file_init_dax( · line 558
Quoted 2 lines
fs/fuse/famfs.c +0
-0/* Publish the famfs metadata on fi->famfs_meta */ inode_lock (inode);
[ ... ]
While reviewing famfs_fuse_meta_alloc() in the same file:
Quoted 7 lines
fs/fuse/famfs.c +0
-0for (i =0 ; i < fmh->nextents; i++) {meta->se[i].dev_index = se_in[i].se_devindex; meta->se[i].ext_offset = se_in[i].se_offset; meta->se[i].ext_len = se_in[i].se_len; /* Record bitmap of referenced daxdev indices */ meta->dev_bitmap |= (1 << meta->se[i].dev_index);
[Severity: High]
Is it possible for the untrusted device index from the FUSE payload (se_devindex) to cause undefined behavior here?
If a FUSE server provides an se_devindex >= 31, the expression 1 << meta->se[i].dev_index shifts a 32-bit signed integer by an unvalidated amount before famfs_check_ext_alignment() validates the index.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5