Skip to contentSkip to conversation

Counters in df, du

PATCH In review Alexey Vlasov 4 participants 11 messages Jul 13, 2009 – Sep 3, 2009 VFS

lore
Share & tools
lore
#0 lore raw

Hi.

Periodically there is an overflow on /tmp on my boxes, and I can't find the reaason why. May be it isn't a bug in tmpfs and you can help me find the reason, and if it is an error, I hope you will help me to eliminate it.

I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.

Well now I see this picture on one of my boxes. Below I post the results of how I was trying to find what was /tmp full of.

# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 4.1G  3.3G  762M  82% /tmp
# du -hs /tmp
518M    /tmp
# lsof /tmp | awk '{if ($7 != 0) print$0}'
(in attach)

I still don't see any files taking 3G. Now I know only one solution mount/remount /tmp.

Signature · 1 line
BRGDS. Alexey Vlasov.
# lsof /tmp | awk '{if ($7 != 0) print$0}'
COMMAND     PID                           USER   FD   TYPE             DEVICE  SIZE      NODE NAME
authdaemo  6555                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6558                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6559                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6560                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6561                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6562                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
mysqld    22905                          mysql    6u   REG               0,13    20 663938322 /tmp/ib1oZTSZ (deleted)
mysqld    22905                          mysql    8u   REG               0,13 84773 663938424 /tmp/ibRcarbD (deleted)
#1 lore raw
Quoted 3 lines · Alexey Vlasov

I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.

You can check for that by doing ls -l /proc/*/fd | grep deleted

It won't tell you the file size though.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#2 lore raw
Quoted 8 lines · Andi Kleen
Quoted 3 lines · Alexey Vlasov

I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.

You can check for that by doing ls -l /proc/*/fd | grep deleted

It won't tell you the file size though.

There comes a big list, like this:

lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) [cut]
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)

But you can see the file date, these are new files and by next search you won't see them any more.

The most interesting is that the /tmp size constantly grows.

Signature · 1 line
BRGDS. Alexey Vlasov.
#3 lore raw

PATCH 1/1 Report file size for deleted files; was Re: Counters in df, du

Quoted 1 line

But you can see the file date, these are new files and by next search

I appended a patch that reports the size.

-Andi

---

VFS: Report file size for deleted files in /proc

It's useful to know how big deleted files are, in case a file system is filled up unexpectedly and the admin wants to track down which process is the culprit. This patch reports the file size in addition to (deleted) for /proc/*/fd links.

This modifies a public interface, but it's reasonable to assume that noone relies on the particular format here.

---
 fs/dcache.c |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
fs/dcache.c +12 -2
Index: linux-2.6.31-rc1-ak/fs/dcache.c
===================================================================
@@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,
 
 	spin_lock(&vfsmount_lock);
 	prepend(&end, &buflen, "\0", 1);
-	if (d_unlinked(dentry) &&
-		(prepend(&end, &buflen, " (deleted)", 10) != 0))
+	if (d_unlinked(dentry)) {
+		char buf[40];
+		int n;
+		if (dentry->d_inode) { /* check needed? */
+			n = snprintf(buf, sizeof buf, " (deleted; size %lluK)",
+					dentry->d_inode->i_size >> 10);
+		} else {
+			strcpy(buf, " (deleted)");
+			n = 10;
+		}
+		if (prepend(&end, &buflen, buf, n) != 0)
 			goto Elong;
+	}
 
 	if (buflen < 1)
 		goto Elong;
Signature · 1 line
[email protected] -- Speaking for myself only.
#4 lore raw

Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du

Quoted 3 lines · Andi Kleen
Quoted 1 line

But you can see the file date, these are new files and by next search

I appended a patch that reports the size.

You can get the size already by using stat instead of lstat, ie. use ls -lL.

Andreas.

Signature · 3 lines
Andreas Schwab, [email protected]
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."
#5 lore raw

Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du

Quoted 7 lines · Andreas Schwab
Quoted 3 lines · Andi Kleen
Quoted 1 line

But you can see the file date, these are new files and by next search

I appended a patch that reports the size.

You can get the size already by using stat instead of lstat, ie. use ls -lL.

True, on the other hand you need two passes because that doesn't give you file names. So the patch would be still more convenient.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#6 lore raw

Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du

In reply to Andi Kleen

Hi Andi Kleen.

Patch can hardly hellp, because there is another problem. Proc shows neither the size of deleted files, nor what he's full with /tmp.

Quoted 53 lines · Andi Kleen
Quoted 1 line

But you can see the file date, these are new files and by next search

I appended a patch that reports the size.

-Andi

---

VFS: Report file size for deleted files in /proc

It's useful to know how big deleted files are, in case a file system is filled up unexpectedly and the admin wants to track down which process is the culprit. This patch reports the file size in addition to (deleted) for /proc/*/fd links.

This modifies a public interface, but it's reasonable to assume that noone relies on the particular format here.

---
 fs/dcache.c |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

Index: linux-2.6.31-rc1-ak/fs/dcache.c
===================================================================

fs/dcache.c +12 -2
@@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,
 
 	spin_lock(&vfsmount_lock);
 	prepend(&end, &buflen, "\0", 1);
-	if (d_unlinked(dentry) &&
-		(prepend(&end, &buflen, " (deleted)", 10) != 0))
+	if (d_unlinked(dentry)) {
+		char buf[40];
+		int n;
+		if (dentry->d_inode) { /* check needed? */
+			n = snprintf(buf, sizeof buf, " (deleted; size %lluK)",
+					dentry->d_inode->i_size >> 10);
+		} else {
+			strcpy(buf, " (deleted)");
+			n = 10;
+		}
+		if (prepend(&end, &buflen, buf, n) != 0)
 			goto Elong;
+	}
 
 	if (buflen < 1)
 		goto Elong;
Signature · 2 lines
-- 
[email protected] -- Speaking for myself only.
Signature · 1 line
BRGDS. Alexey Vlasov.
#7 lore raw
In reply to Alexey Vlasov

2009/7/13 Alexey Vlasov <[email protected]>:

Quoted 20 lines · Andi Kleen
Quoted 8 lines
Quoted 3 lines · Alexey Vlasov

I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.

You can check for that by doing ls -l /proc/*/fd | grep deleted

It won't tell you the file size though.

There comes a big list, like this:

lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) [cut]
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)

Quoted 4 lines

But you can see the file date, these are new files and by next search you won't see them any more.

The most interesting is that the /tmp size constantly grows.

Try:

lsof -n | grep deleted

This will show the process holding them more easily, and the file size (in the seventh column). It won't show the date they were opened though.

#8 lore raw
Quoted 9 lines · Bryan Donlan
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines · Alexey Vlasov

I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.

You can check for that by doing ls -l /proc/*/fd | grep deleted

It won't tell you the file size though.

Quoted 7 lines
Quoted 5 lines

There comes a big list, like this:

lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)

The most interesting is that the /tmp size constantly grows.

lsof -n | grep deleted

# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'

mysqld 22905 mysql 6u REG
0,13 20 663938322 /tmp/ib1oZTSZ (deleted)
mysqld 22905 mysql 8u REG
0,13 84773 663938424 /tmp/ibRcarbD (deleted)

# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 4.1G  3.4G  630M  85% /tmp

It will soon be up to 100% full and I'll have to remount.

Signature · 1 line
BRGDS. Alexey Vlasov.
#9 lore raw

2009/7/13 Alexey Vlasov <[email protected]>:

Quoted 25 lines · Bryan Donlan
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines · Alexey Vlasov

I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.

You can check for that by doing ls -l /proc/*/fd | grep deleted

It won't tell you the file size though.

Quoted 7 lines
Quoted 5 lines

There comes a big list, like this:

lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)

The most interesting is that the /tmp size constantly grows.

lsof -n | grep deleted

# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'

mysqld    22905                          mysql    6u   REG 0,13    20 663938322 /tmp/ib1oZTSZ (deleted) mysqld    22905                          mysql    8u   REG 0,13 84773 663938424 /tmp/ibRcarbD (deleted)

Quoted 5 lines
# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 4.1G  3.4G  630M  85% /tmp

It will soon be up to 100% full and I'll have to remount.

Are those the only two results? That should only be adding up to ~84kb of space...

#10 lore raw
Quoted 8 lines · Bryan Donlan

2009/7/13 Alexey Vlasov <[email protected]>:

Quoted 7 lines
# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'

mysqld    22905                          mysql    6u   REG 0,13    20 663938322 /tmp/ib1oZTSZ (deleted) mysqld    22905                          mysql    8u   REG 0,13 84773 663938424 /tmp/ibRcarbD (deleted)

Quoted 6 lines
Quoted 3 lines
# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 4.1G  3.4G  630M  85% /tmp

Are those the only two results? That should only be adding up to ~84kb of space...

Yes, only two lines where the 7-th column isn't equal to 0

Signature · 1 line
BRGDS. Alexey Vlasov.