Skip to contentSkip to conversation

proc: report open files as size in stat() for /proc/pid/fd

PATCH Superseded Ivan Babrou 2 participants 3 messages Sep 20, 2022 – Sep 21, 2022

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 proc: report open files as size in stat() for /proc/pid/fd

Many monitoring tools include open file count as a metric. Currently the only way to get this number is to enumerate the files in /proc/pid/fd.

The problem with the current approach is that it does many things people generally don't care about when they need one number for a metric. In our tests for cadvisor, which reports open file counts per cgroup, we observed that reading the number of open files is slow. Out of 35.23% of CPU time spent in proc_readfd_common, we see 29.43% spent in proc_fill_cache, which is responsible for filling dentry info. Some of this extra time is spinlock contention, but it's a contention for the lock we don't want to take to begin with.

We considered putting the number of open files in /proc/pid/status. Unfortunately, counting the number of fds involves iterating the open_files bitmap, which has a linear complexity in proportion with the number of open files (bitmap slots really, but it's close). We don't want to make /proc/pid/status any slower, so instead we put this info in /proc/pid/fd as a size member of the stat syscall result. Previously the reported number was zero, so there's very little risk of breaking anything, while still providing a somewhat logical way to count the open files with a fallback if it's zero.

RFC for this patch included iterating open fds under RCU. Thanks to Frank Hofmann for the suggestion to use the bitmap instead.

Previously:

```
$ sudo stat /proc/1/fd | head -n2
  File: /proc/1/fd
  Size: 0         	Blocks: 0          IO Block: 1024   directory
```

With this patch:

```
$ sudo stat /proc/1/fd | head -n2
  File: /proc/1/fd
  Size: 65        	Blocks: 0          IO Block: 1024   directory
```

Correctness check:

```
$ sudo ls /proc/1/fd | wc -l
65
```

I added the docs for /proc/<pid>/fd while I'm at it.

---
 Documentation/filesystems/proc.rst | 17 ++++++++++++++
 fs/proc/fd.c                       | 36 ++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
Documentation/filesystems/proc.rst +17 -0
index e7aafc82be99..394548d26187 100644
@@ -47,6 +47,7 @@ fixes/update part 1.1  Stefani Seibold <[email protected]>    June 9 2009
   3.10  /proc/<pid>/timerslack_ns - Task timerslack value
   3.11	/proc/<pid>/patch_state - Livepatch patch operation state
   3.12	/proc/<pid>/arch_status - Task architecture specific information
+  3.13  /proc/<pid>/fd - List of symlinks to open files
 
   4	Configuring procfs
   4.1	Mount options
@@ -2145,6 +2146,22 @@ AVX512_elapsed_ms
   the task is unlikely an AVX512 user, but depends on the workload and the
   scheduling scenario, it also could be a false negative mentioned above.
 
+3.13 /proc/<pid>/fd - List of symlinks to open files
+-------------------------------------------------------
+This directory contains symbolic links which represent open files
+the process is maintaining.  Example output::
+
+  lr-x------ 1 root root 64 Sep 20 17:53 0 -> /dev/null
+  l-wx------ 1 root root 64 Sep 20 17:53 1 -> /dev/null
+  lrwx------ 1 root root 64 Sep 20 17:53 10 -> 'socket:[12539]'
+  lrwx------ 1 root root 64 Sep 20 17:53 11 -> 'socket:[12540]'
+  lrwx------ 1 root root 64 Sep 20 17:53 12 -> 'socket:[12542]'
+
+The number of open files for the process is stored in 'size' member
+of stat() output for /proc/<pid>/fd for fast access.
+-------------------------------------------------------
+
+
 Chapter 4: Configuring procfs
 =============================
 
fs/proc/fd.c +36 -0
index 913bef0d2a36..c3c4bf2939a7 100644
@@ -279,6 +279,26 @@ static int proc_readfd_common(struct file *file, struct dir_context *ctx,
 	return 0;
 }
 
+static int proc_readfd_count(struct inode *inode)
+{
+	struct task_struct *p = get_proc_task(inode);
+	struct fdtable *fdt;
+	unsigned int i, size, open_fds = 0;
+
+	if (!p)
+		return -ENOENT;
+
+	if (p->files) {
+		fdt = files_fdtable(p->files);
+		size = fdt->max_fds;
+
+		for (i = size / BITS_PER_LONG; i > 0;)
+			open_fds += hweight64(fdt->open_fds[--i]);
+	}
+
+	return open_fds;
+}
+
 static int proc_readfd(struct file *file, struct dir_context *ctx)
 {
 	return proc_readfd_common(file, ctx, proc_fd_instantiate);
@@ -319,9 +339,25 @@ int proc_fd_permission(struct user_namespace *mnt_userns,
 	return rv;
 }
 
+static int proc_fd_getattr(struct user_namespace *mnt_userns,
+			const struct path *path, struct kstat *stat,
+			u32 request_mask, unsigned int query_flags)
+{
+	struct inode *inode = d_inode(path->dentry);
+
+	generic_fillattr(&init_user_ns, inode, stat);
+
+	/* If it's a directory, put the number of open fds there */
+	if (S_ISDIR(inode->i_mode))
+		stat->size = proc_readfd_count(inode);
+
+	return 0;
+}
+
 const struct inode_operations proc_fd_inode_operations = {
 	.lookup		= proc_lookupfd,
 	.permission	= proc_fd_permission,
+	.getattr	= proc_fd_getattr,
 	.setattr	= proc_setattr,
 };
 
Signature · 1 line
2.37.2
#1 lore raw

From: Ivan Babrou

Quoted 1 line

Sent: 20 September 2022 20:06

...

Quoted 11 lines
fs/proc/fd.c +10 -0
+static int proc_readfd_count(struct inode *inode)
+{
+	struct task_struct *p = get_proc_task(inode);
+	struct fdtable *fdt;
+	unsigned int i, size, open_fds = 0;
+
+	if (!p)
+		return -ENOENT;
+
+	if (p->files) {

fs/proc/fd.c static int proc_readfd_common(struct file *file, struct dir_context *ctx, · line 301

Quoted 10 lines
fs/proc/fd.c +10 -0
+		fdt = files_fdtable(p->files);
+		size = fdt->max_fds;
+
+		for (i = size / BITS_PER_LONG; i > 0;)
+			open_fds += hweight64(fdt->open_fds[--i]);
+	}
+
+	return open_fds;
+}
+

Doesn't that need (at least) rcu protection? There might also be issues reading p->files twice.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#2 lore raw
Quoted 5 lines · David Laight

From: Ivan Babrou

Quoted 1 line

Sent: 20 September 2022 20:06

...

Quoted 1 line
Quoted 16 lines
Quoted 6 lines
fs/proc/fd.c +6 -0
+static int proc_readfd_count(struct inode *inode)
+{
+     struct task_struct *p = get_proc_task(inode);
+     struct fdtable *fdt;
+     unsigned int i, size, open_fds = 0;
+

fs/proc/fd.c static int proc_readfd_common(struct file *file, struct dir_context *ctx, · line 297

Quoted 10 lines
fs/proc/fd.c +10 -0
+     if (!p)
+             return -ENOENT;
+
+     if (p->files) {
+             fdt = files_fdtable(p->files);
+             size = fdt->max_fds;
+
+             for (i = size / BITS_PER_LONG; i > 0;)
+                     open_fds += hweight64(fdt->open_fds[--i]);
+     }

I'm missing put_task_struct(p) here.

Quoted 6 lines

fs/proc/fd.c static int proc_readfd_common(struct file *file, struct dir_context *ctx, · line 301

Quoted 4 lines
fs/proc/fd.c +4 -0
+
+     return open_fds;
+}
+

Doesn't that need (at least) rcu protection?

Should I enclose the "if" in rcu_read_lock() / rcu_read_unlock()?

files_fdtable() is this:

* https://elixir.bootlin.com/linux/v6.0-rc6/source/include/linux/fdtable.h#L77

#define files_fdtable(files) \
    rcu_dereference_check_fdtable((files), (files)->fdt)

And rcu_dereference_check_fdtable() is

#define rcu_dereference_check_fdtable(files, fdtfd) \
rcu_dereference_check((fdtfd), lockdep_is_held(&(files)->file_lock))

I definitely need some help with locking here.

Quoted 1 line

There might also be issues reading p->files twice.

This block for reading twice:

if (p->files) {
    fdt = files_fdtable(p->files);

Already exists in fs/proc/array.c in task_state():

* https://elixir.bootlin.com/linux/v6.0-rc6/source/fs/proc/array.c#L173

There's task_lock(p) there, so maybe that's why it's allowed.

Should I run files_fdtable(p->files) unconditionally and then check the result instead? I'm happy to change it to something else if you can tell me what it should be.

If there are kernel options I should enable for testing this, I'd be happy to hear them. So far we've tried running with KASAN with no issues.