Skip to contentSkip to conversation

Generic per-mount io stats

RFC PATCH In review Amir Goldstein 1 participant 5 messages Jan 7, 2021 – Jan 8, 2021 VFS

lore
Share & tools
lore
#0 lore raw

cover Generic per-mount io stats

Miklos,

I was trying to address the lack of iostat report for non-blockdev filesystems such as overlayfs and fuse.

NFS has already addressed this with it own custom stats collection, which is displayed in /proc/<pid>/mountstats.

When looking at the options, I found that a generic solution is quite simple and could serve all filesystems that opt-in to use it.

This short patch set results in the following mountstats example report:

device overlay mounted on /mnt with fstype overlay

	times: 125 153
	rchar: 12
	wchar: 0
	syscr: 2
	syscw: 0

The choise to collect and report io stats by mount and not by sb is quite arbitrary, because it was quite easy to implement and is natural to the existing mountstats proc file.

I used the arbirtaty flag FS_USERNS_MOUNT as an example for a way for filesystem to opt-in to mount io stats, but it could be either an FS_ SB_ or MNT_ flag. I do not anticipate shortage of opinions on this matter.

As for performance, the io accounting hooks are the existing hooks for task io accounting. mount io stats add a dereference to mnt_pcp for the filesystems that opt-in and one per-cpu var update. The dereference to mnt_sb->s_type->fs_flags is temporary as we will probably want to use an MNT_ flag, whether kernel internal or user controlled.

What do everyone think about this?

Al,

did I break any subtle rules of the vfs?

Thanks,
Amir.

Amir Goldstein (3):
fs: add iostats counters to struct mount
fs: collect per-mount io stats
fs: report per-mount io stats

 fs/Kconfig          |  9 +++++
 fs/mount.h          | 54 ++++++++++++++++++++++++++++
 fs/namespace.c      | 19 ++++++++++
 fs/proc_namespace.c | 13 +++++++
 fs/read_write.c     | 87 ++++++++++++++++++++++++++++++++-------------
 5 files changed, 158 insertions(+), 24 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH 1/3 fs: add iostats counters to struct mount

With config MOUNT_IO_STATS, add an array of counters to struct mnt_pcp that will be used to collect I/O statistics.

---

Please note that the dependency on SMP is just for the RFC.

Thanks,
Amir.

 fs/Kconfig     |  9 +++++++++
 fs/mount.h     | 32 ++++++++++++++++++++++++++++++++
 fs/namespace.c | 17 +++++++++++++++++
 3 files changed, 58 insertions(+)
fs/Kconfig +9 -0
index aa4c12282301..7473bdf4bbfb 100644
@@ -15,6 +15,15 @@ config VALIDATE_FS_PARSER
 	  Enable this to perform validation of the parameter description for a
 	  filesystem when it is registered.
 
+config FS_MOUNT_STATS
+	bool "Enable per-mount I/O statistics"
+	depends on SMP
+	help
+	  Enable this to allow collecting per-mount I/O statistics and display
+	  them in /proc/<pid>/mountstats.
+
+	  Say N if unsure.
+
 if BLOCK
 
 config FS_IOMAP
fs/mount.h +32 -0
index ce6c376e0bc2..2bf0df64ded5 100644
@@ -24,9 +24,25 @@ struct mnt_namespace {
 	unsigned int		pending_mounts;
 } __randomize_layout;
 
+/* Similar to task_io_accounting members */
+enum {
+	MNTIOS_CHARS_RD,	/* bytes read via syscalls */
+	MNTIOS_CHARS_WR,	/* bytes written via syscalls */
+	MNTIOS_SYSCALLS_RD,	/* # of read syscalls */
+	MNTIOS_SYSCALLS_WR,	/* # of write syscalls */
+	_MNTIOS_COUNTERS_NUM
+};
+
+struct mnt_iostats {
+	s64 counter[_MNTIOS_COUNTERS_NUM];
+};
+
 struct mnt_pcp {
 	int mnt_count;
 	int mnt_writers;
+#ifdef CONFIG_FS_MOUNT_STATS
+	struct mnt_iostats iostats;
+#endif
 };
 
 struct mountpoint {
@@ -158,3 +174,19 @@ static inline bool is_anon_ns(struct mnt_namespace *ns)
 }
 
 extern void mnt_cursor_del(struct mnt_namespace *ns, struct mount *cursor);
+
+static inline void mnt_iostats_counter_inc(struct mount *mnt, int id)
+{
+#ifdef CONFIG_FS_MOUNT_STATS
+	this_cpu_inc(mnt->mnt_pcp->iostats.counter[id]);
+#endif
+}
+
+static inline void mnt_iostats_counter_add(struct mount *mnt, int id, s64 n)
+{
+#ifdef CONFIG_FS_MOUNT_STATS
+	this_cpu_add(mnt->mnt_pcp->iostats.counter[id], n);
+#endif
+}
+
+extern s64 mnt_iostats_counter_read(struct mount *mnt, int id);
fs/namespace.c +17 -0
index d2db7dfe232b..04b35dfcc71f 100644
@@ -283,6 +283,23 @@ static unsigned int mnt_get_writers(struct mount *mnt)
 #endif
 }
 
+s64 mnt_iostats_counter_read(struct mount *mnt, int id)
+{
+	s64 count = 0;
+#ifdef CONFIG_FS_MOUNT_STATS
+	/*
+	 * MOUNT_STATS depends on SMP.
+	 * Should be trivial to implement for !SMP if anyone cares...
+	 */
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		count += per_cpu_ptr(mnt->mnt_pcp, cpu)->iostats.counter[id];
+	}
+#endif
+	return count;
+}
+
 static int mnt_is_readonly(struct vfsmount *mnt)
 {
 	if (mnt->mnt_sb->s_readonly_remount)
Signature · 1 line
2.25.1
#2 lore raw

RFC PATCH 2/3 fs: collect per-mount io stats

In reply to Amir Goldstein

Replace task io account helpers with wrappers that may also collect per-mount stats.

Currently, just for example, stats are collected for mounts of filesystems with flag FS_USERNS_MOUNT.

---

I used the arbirtaty flag FS_USERNS_MOUNT as an example for a way for filesystem to opt-in to mount io stats, but it could be either an FS_ SB_ or MNT_ flag. I do not anticipate shortage of opinions on this matter.

As for performance, the io accounting hooks are the existing hooks for task io accounting. mount io stats add a dereference to mnt_pcp for the filesystems that opt-in and one per-cpu var update. The dereference to mnt_sb->s_type->fs_flags is temporary as we will probably want to use an MNT_ flag, whether kernel internal or user controlled.

Thanks,
Amir.

 fs/mount.h      | 21 ++++++++++++
 fs/read_write.c | 87 +++++++++++++++++++++++++++++++++++--------------
 2 files changed, 84 insertions(+), 24 deletions(-)
fs/mount.h +21 -0
index 2bf0df64ded5..81db83c36140 100644
@@ -175,6 +175,27 @@ static inline bool is_anon_ns(struct mnt_namespace *ns)
 
 extern void mnt_cursor_del(struct mnt_namespace *ns, struct mount *cursor);
 
+static inline bool mnt_has_stats(struct vfsmount *mnt)
+{
+#ifdef CONFIG_FS_MOUNT_STATS
+	/* Just for example. Should this be an FS_ SB_ or MNT_ flag? */
+	return (mnt->mnt_sb->s_type->fs_flags & FS_USERNS_MOUNT);
+#else
+	return false;
+#endif
+}
+
+static inline struct mount *file_mnt_has_stats(struct file *file)
+{
+#ifdef CONFIG_FS_MOUNT_STATS
+	struct vfsmount *mnt = file->f_path.mnt;
+
+	if (mnt_has_stats(mnt))
+		return real_mount(mnt);
+#endif
+	return NULL;
+}
+
 static inline void mnt_iostats_counter_inc(struct mount *mnt, int id)
 {
 #ifdef CONFIG_FS_MOUNT_STATS
fs/read_write.c +63 -24

172 lines not on this page · raw message

Signature · 1 line
2.25.1
#3 lore raw

RFC PATCH 3/3 fs: report per-mount io stats

In reply to Amir Goldstein

Show optional collected per-mount io stats in /proc/<pid>/mountstats for filesystems that do not implement their own show_stats() method.

---

See following snippet from mountstats example report:

device overlay mounted on /mnt with fstype overlay

	times: 125 153
	rchar: 12
	wchar: 0
	syscr: 2
	syscw: 0

Thanks,
Amir.

 fs/mount.h          |  1 +
 fs/namespace.c      |  2 ++
 fs/proc_namespace.c | 13 +++++++++++++
 3 files changed, 16 insertions(+)
fs/mount.h +1 -0
index 81db83c36140..1f262892a6ed 100644
@@ -91,6 +91,7 @@ struct mount {
 	int mnt_id;			/* mount identifier */
 	int mnt_group_id;		/* peer group identifier */
 	int mnt_expiry_mark;		/* true if marked for expiry */
+	time64_t mnt_time;		/* time of mount */
 	struct hlist_head mnt_pins;
 	struct hlist_head mnt_stuck_children;
 } __randomize_layout;
fs/namespace.c +2 -0
index 04b35dfcc71f..3a91234e5fd0 100644
@@ -198,6 +198,8 @@ static struct mount *alloc_vfsmnt(const char *name)
 		mnt->mnt_count = 1;
 		mnt->mnt_writers = 0;
 #endif
+		/* For proc/<pid>/mountstats */
+		mnt->mnt_time = ktime_get_seconds();
 
 		INIT_HLIST_NODE(&mnt->mnt_hash);
 		INIT_LIST_HEAD(&mnt->mnt_child);
fs/proc_namespace.c +13 -0
index eafb75755fa3..34aea7f3f550 100644
@@ -229,6 +229,19 @@ static int show_vfsstat(struct seq_file *m, struct vfsmount *mnt)
 	if (sb->s_op->show_stats) {
 		seq_putc(m, ' ');
 		err = sb->s_op->show_stats(m, mnt_path.dentry);
+	} else if (mnt_has_stats(mnt)) {
+		/* Similar to /proc/<pid>/io */
+		seq_printf(m, "\n"
+			   "\ttimes: %lld %lld\n"
+			   "\trchar: %lld\n"
+			   "\twchar: %lld\n"
+			   "\tsyscr: %lld\n"
+			   "\tsyscw: %lld\n",
+			   r->mnt_time, ktime_get_seconds(),
+			   mnt_iostats_counter_read(r, MNTIOS_CHARS_RD),
+			   mnt_iostats_counter_read(r, MNTIOS_CHARS_WR),
+			   mnt_iostats_counter_read(r, MNTIOS_SYSCALLS_RD),
+			   mnt_iostats_counter_read(r, MNTIOS_SYSCALLS_WR));
 	}
 
 	seq_putc(m, '\n');
Signature · 1 line
2.25.1
#4 lore raw
In reply to Amir Goldstein
Quoted 36 lines · Amir Goldstein

Miklos,

I was trying to address the lack of iostat report for non-blockdev filesystems such as overlayfs and fuse.

NFS has already addressed this with it own custom stats collection, which is displayed in /proc/<pid>/mountstats.

When looking at the options, I found that a generic solution is quite simple and could serve all filesystems that opt-in to use it.

This short patch set results in the following mountstats example report:

device overlay mounted on /mnt with fstype overlay

        times: 125 153
        rchar: 12
        wchar: 0
        syscr: 2
        syscw: 0

The choise to collect and report io stats by mount and not by sb is quite arbitrary, because it was quite easy to implement and is natural to the existing mountstats proc file.

I used the arbirtaty flag FS_USERNS_MOUNT as an example for a way for filesystem to opt-in to mount io stats, but it could be either an FS_ SB_ or MNT_ flag. I do not anticipate shortage of opinions on this matter.

As for performance, the io accounting hooks are the existing hooks for task io accounting. mount io stats add a dereference to mnt_pcp for the filesystems that opt-in and one per-cpu var update. The dereference to mnt_sb->s_type->fs_flags is temporary as we will probably want to use an MNT_ flag, whether kernel internal or user controlled.

Quoted 6 lines

What do everyone think about this?

Al,

did I break any subtle rules of the vfs?

That is besides dereferencing a NULL file pointer when getting EBADF in p/readv/writev...

Thanks,
Amir.