shmem: Allow userspace monitoring of tmpfs for lack of space.
Subsystems (3)
cover shmem: Allow userspace monitoring of tmpfs for lack of space.
When provisioning containerized applications, multiple very small tmpfs are used, for which one cannot always predict the proper file system size ahead of time. We want to be able to reliably monitor filesystems for ENOSPC errors, without depending on the application being executed reporting the ENOSPC after a failure. It is also not enough to watch statfs since that information might be ephemeral (say the application recovers by deleting data, the issue can get lost). For this use case, it is also interesting to differentiate IO errors caused by lack of virtual memory from lack of FS space.
This patch exposes two counters on sysfs that log the two conditions that are interesting to observe for container provisioning. They are recorded per tmpfs superblock, and can be polled by a monitoring application.
I proposed a more general approach [1] using fsnotify, but considering the specificity of this use-case, people agreed it seems that a simpler solution in sysfs is more than enough.
[1] https://lore.kernel.org/linux-mm/[email protected]/T/#mee338d25b0e1e07cbe0861f9a5ca8cc439b3edb8
Gabriel Krisman Bertazi (3):
shmem: Keep track of out-of-memory and out-of-space errors
shmem: Introduce /sys/fs/tmpfs support
shmem: Expose space and accounting error count
Documentation/ABI/testing/sysfs-fs-tmpfs | 13 +++ include/linux/shmem_fs.h | 7 ++ mm/shmem.c | 103 ++++++++++++++++++++++- 3 files changed, 121 insertions(+), 2 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-fs-tmpfs
Signature · 1 line
2.35.1
PATCH 1/3 shmem: Keep track of out-of-memory and out-of-space errors
Keep a per-sb counter of failed shmem allocations for ENOMEM/ENOSPC to be reported on sysfs. The sysfs support is done separately on a later patch.
--- include/linux/shmem_fs.h | 3 +++ mm/shmem.c | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-)
include/linux/shmem_fs.h +3 -0
@@ -44,6 +44,9 @@ struct shmem_sb_info {spinlock_t shrinklist_lock;/* Protects shrinklist */ struct list_head shrinklist;/* List of shinkable inodes */ unsigned long shrinklist_len;/* Length of shrinklist */ + +unsigned long acct_errors; +unsigned long space_errors; };static inline struct shmem_inode_info *SHMEM_I (struct inode *inode)
mm/shmem.c +4 -1
@@ -212,8 +212,10 @@ static inline bool shmem_inode_acct_block(struct inode *inode, long pages)struct shmem_inode_info *info =SHMEM_I (inode);struct shmem_sb_info *sbinfo =SHMEM_SB (inode->i_sb); -if (shmem_acct_block (info->flags, pages)) +if (shmem_acct_block (info->flags, pages)) { + sbinfo->acct_errors +=1 ;return false; + }if (sbinfo->max_blocks) { if (percpu_counter_compare(&sbinfo->used_blocks,
@@ -225,6 +227,7 @@ static inline bool shmem_inode_acct_block(struct inode *inode, long pages)return true; unacct: + sbinfo->space_errors +=1 ;shmem_unacct_blocks (info->flags, pages);return false; }
Signature · 1 line
2.35.1
PATCH 2/3 shmem: Introduce /sys/fs/tmpfs support
In reply to Gabriel Krisman BertaziIn order to expose tmpfs statistics on sysfs, add the boilerplate code to create the /sys/fs/tmpfs structure. Other filesystems usually do /sys/fs/<fs>/<disk>, but since this is a nodev filesystem, I'm proposing to use fsid as <disk>.
This takes care of not exposing SB_NOUSER mounts. I don't think we have a usecase for showing them and, since they don't appear elsewhere, they might be confusing to users.
--- include/linux/shmem_fs.h | 4 +++ mm/shmem.c | 73 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 76 insertions(+), 1 deletion(-)
include/linux/shmem_fs.h +4 -0
@@ -47,6 +47,10 @@ struct shmem_sb_info {unsigned long acct_errors;unsigned long space_errors; + +/* sysfs */ +struct kobject s_kobj;/* /sys/fs/tmpfs/<uuid> */ +struct completion s_kobj_unregister; };static inline struct shmem_inode_info *SHMEM_I (struct inode *inode)
mm/shmem.c +72 -1
113 lines not on this page · raw message
Signature · 1 line
2.35.1
Re: [PATCH 2/3] shmem: Introduce /sys/fs/tmpfs support
[+fsdevel please CC on next versions]
Quoted 5 lines · Gabriel Krisman Bertazi
In order to expose tmpfs statistics on sysfs, add the boilerplate code to create the
/sys/fs/tmpfsstructure. Other filesystems usually do /sys/fs/<fs>/<disk>, but since this is a nodev filesystem, I'm proposing to use fsid as <disk>.
I am proposing st_dev minor.
Quoted 11 lines
This takes care of not exposing
SB_NOUSERmounts. I don't think we have a usecase for showing them and, since they don't appear elsewhere, they might be confusing to users.Signed-off-by Gabriel Krisman Bertazi <[email protected]>--- include/linux/shmem_fs.h | 4 +++ mm/shmem.c | 73 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 76 insertions(+), 1 deletion(-)
[...]
mm/shmem.c static int shmem_show_options(struct seq_file *seq, struct dentry *root) · line 3628
Quoted 9 lines
mm/shmem.c +9
-0+static int shmem_register_sysfs (struct super_block *sb) +{ +int err; +struct shmem_sb_info *sbinfo =SHMEM_SB (sb); +__kernel_fsid_t fsid =uuid_to_fsid (sb->s_uuid.b); + +init_completion (&sbinfo->s_kobj_unregister); + err =kobject_init_and_add (&sbinfo->s_kobj, &tmpfs_sb_ktype, shmem_root, +"%x%x" , fsid.val[0 ], fsid.val[1 ]);
uuid (and fsid) try to be unique across tmpfs instances from different times. You don't need that.
I think you'd rather use s_dev (minor number) which is unique among all tmpfs instances at a given time and also much easier from user scripts to read from (e.g. stat or /proc/self/mountinfo).
That's btw the same number is used as an entry in /sys/fs/fuse/connections (fusectl pseudo fs).
Thanks,
Amir.
Re: [PATCH 2/3] shmem: Introduce /sys/fs/tmpfs support
Quoted 16 lines · Amir Goldstein
mm/shmem.c static int shmem_show_options(struct seq_file *seq, struct dentry *root) · line 3628
Quoted 9 lines
mm/shmem.c +9
-0+static int shmem_register_sysfs (struct super_block *sb) +{ +int err; +struct shmem_sb_info *sbinfo =SHMEM_SB (sb); +__kernel_fsid_t fsid =uuid_to_fsid (sb->s_uuid.b); + +init_completion (&sbinfo->s_kobj_unregister); + err =kobject_init_and_add (&sbinfo->s_kobj, &tmpfs_sb_ktype, shmem_root, +"%x%x" , fsid.val[0 ], fsid.val[1 ]);uuid (and fsid) try to be unique across tmpfs instances from different times. You don't need that.
I think you'd rather uses_dev(minor number) which is unique among all tmpfs instances at a given time and also much easier from user scripts to read from (e.g. stat or/proc/self/mountinfo).
Quoted 2 lines
That's btw the same number is used as an entry in
/sys/fs/fuse/connections(fusectl pseudo fs).
Hi Amir, thanks for the review.
Sounds good. I will follow up with a new version that uses MINOR(sb->s_dev).
Thank you,
Signature · 1 line
Gabriel Krisman Bertazi
PATCH 3/3 shmem: Expose space and accounting error count
In reply to Gabriel Krisman BertaziExposing these shmem counters through sysfs is particularly useful for container provisioning, to allow administrators to differentiate between insufficiently provisioned fs size vs. running out of memory.
--- Documentation/ABI/testing/sysfs-fs-tmpfs | 13 ++++++++++++ mm/shmem.c | 25 ++++++++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-fs-tmpfs
Documentation/ABI/testing/sysfs-fs-tmpfs new file +13 -0
@@ -0,0 +1,13 @@ +What: /sys/fs/tmpfs/<disk>/acct_errors +Date: March 2022 +Contact: "Gabriel Krisman Bertazi" <[email protected]> +Description: + Track the number of IO errors caused by lack of memory to + perform the allocation of a tmpfs block. + +What: /sys/fs/tmpfs/<disk>/space_errors +Date: March 2022 +Contact: "Gabriel Krisman Bertazi" <[email protected]> +Description: + Track the number of IO errors caused by lack of space + in the filesystem to perform the allocation of a tmpfs block.
mm/shmem.c +25 -0
@@ -214,6 +214,7 @@ static inline bool shmem_inode_acct_block(struct inode *inode, long pages)if (shmem_acct_block (info->flags, pages)) { sbinfo->acct_errors +=1 ; +sysfs_notify (&sbinfo->s_kobj,NULL ,"acct_errors" );return false; }
@@ -228,6 +229,7 @@ static inline bool shmem_inode_acct_block(struct inode *inode, long pages) unacct: sbinfo->space_errors +=1 ; +sysfs_notify (&sbinfo->s_kobj,NULL ,"space_errors" );shmem_unacct_blocks (info->flags, pages);return false; }
@@ -3586,10 +3588,33 @@ static int shmem_show_options(struct seq_file *seq, struct dentry *root)#endif /* CONFIG_TMPFS */ #if defined(CONFIG_TMPFS ) && defined(CONFIG_SYSFS ) +static ssize_t acct_errors_show (struct kobject *kobj, +struct kobj_attribute *attr,char *page) +{ +struct shmem_sb_info *sbinfo = + container_of(kobj, struct shmem_sb_info, s_kobj); + +return sysfs_emit (page,"%lu\n" , sbinfo->acct_errors); +} + +static ssize_t space_errors_show (struct kobject *kobj, +struct kobj_attribute *attr,char *page) +{ +struct shmem_sb_info *sbinfo = + container_of(kobj, struct shmem_sb_info, s_kobj); + +return sysfs_emit (page,"%lu\n" , sbinfo->space_errors); +} +#define TMPFS_SB_ATTR_RO (name) \ static struct kobj_attribute tmpfs_sb_attr_##name = __ATTR_RO(name) +TMPFS_SB_ATTR_RO (acct_errors); +TMPFS_SB_ATTR_RO (space_errors); +static struct attribute *tmpfs_attrs[] = { + &tmpfs_sb_attr_acct_errors.attr, + &tmpfs_sb_attr_space_errors.attr,NULL };ATTRIBUTE_GROUPS (tmpfs);
Signature · 1 line
2.35.1