Counters in df, du
Hi.
Periodically there is an overflow on /tmp on my boxes, and I can't find the reaason why. May be it isn't a bug in tmpfs and you can help me find the reason, and if it is an error, I hope you will help me to eliminate it.
I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.
Well now I see this picture on one of my boxes. Below I post the results of how I was trying to find what was /tmp full of.
# df -h /tmp Filesystem Size Used Avail Use% Mounted on tmpfs 4.1G 3.3G 762M 82% /tmp
# du -hs /tmp 518M /tmp
# lsof /tmp | awk '{if ($7 != 0) print$0}'
(in attach)I still don't see any files taking 3G. Now I know only one solution mount/remount /tmp.
Signature · 1 line
BRGDS. Alexey Vlasov.
# lsof /tmp | awk '{if ($7 != 0) print$0}'
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
authdaemo 6555 root 5u unix 0xffff81030b8c2840 281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo 6558 root 5u unix 0xffff81030b8c2840 281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo 6559 root 5u unix 0xffff81030b8c2840 281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo 6560 root 5u unix 0xffff81030b8c2840 281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo 6561 root 5u unix 0xffff81030b8c2840 281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo 6562 root 5u unix 0xffff81030b8c2840 281375180 /var/lib/courier/authdaemon/socket.tmp
mysqld 22905 mysql 6u REG 0,13 20 663938322 /tmp/ib1oZTSZ (deleted)
mysqld 22905 mysql 8u REG 0,13 84773 663938424 /tmp/ibRcarbD (deleted)Quoted 3 lines · Alexey Vlasov
I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.
You can check for that by doing ls -l /proc/*/fd | grep deleted
It won't tell you the file size though.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 8 lines · Andi Kleen
Quoted 3 lines · Alexey Vlasov
I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.
You can check for that by doing ls -l
/proc/*/fd| grep deletedIt won't tell you the file size though.
There comes a big list, like this:
lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted) [cut]
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
But you can see the file date, these are new files and by next search you won't see them any more.
The most interesting is that the /tmp size constantly grows.
Signature · 1 line
BRGDS. Alexey Vlasov.
PATCH 1/1 Report file size for deleted files; was Re: Counters in df, du
Quoted 1 line
But you can see the file date, these are new files and by next search
I appended a patch that reports the size.
-Andi
---
VFS: Report file size for deleted files in /proc
It's useful to know how big deleted files are, in case a file system is filled up unexpectedly and the admin wants to track down which process is the culprit. This patch reports the file size in addition to (deleted) for /proc/*/fd links.
This modifies a public interface, but it's reasonable to assume that noone relies on the particular format here.
--- fs/dcache.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
fs/dcache.c +12 -2
@@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,spin_lock (&vfsmount_lock);prepend (&end, &buflen,"\0" ,1 ); -if (d_unlinked (dentry) && - (prepend (&end, &buflen," (deleted)" ,10 ) !=0 )) +if (d_unlinked (dentry)) { +char buf[40 ]; +int n; +if (dentry->d_inode) {/* check needed? */ + n =snprintf (buf,sizeof buf," (deleted; size %lluK)" , + dentry->d_inode->i_size >>10 ); + }else { +strcpy (buf," (deleted)" ); + n =10 ; + } +if (prepend (&end, &buflen, buf, n) !=0 ) goto Elong; + }if (buflen <1 ) goto Elong;
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du
Quoted 3 lines · Andi Kleen
Quoted 1 line
But you can see the file date, these are new files and by next search
I appended a patch that reports the size.
You can get the size already by using stat instead of lstat, ie. use ls -lL.
Andreas.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84 5EC7 45C6 250E 6F00 984E "And now for something completely different."
Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du
Quoted 7 lines · Andreas Schwab
Quoted 3 lines · Andi Kleen
Quoted 1 line
But you can see the file date, these are new files and by next search
I appended a patch that reports the size.
You can get the size already by using stat instead of lstat, ie. use ls -lL.
True, on the other hand you need two passes because that doesn't give you file names. So the patch would be still more convenient.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du
In reply to Andi KleenHi Andi Kleen.
Patch can hardly hellp, because there is another problem. Proc shows neither the size of deleted files, nor what he's full with /tmp.
Quoted 53 lines · Andi Kleen
Quoted 1 line
But you can see the file date, these are new files and by next search
I appended a patch that reports the size.
-Andi
---
VFS: Report file size for deleted files in /proc
It's useful to know how big deleted files are, in case a file system is filled up unexpectedly and the admin wants to track down which process is the culprit. This patch reports the file size in addition to (deleted) for
/proc/*/fdlinks.This modifies a public interface, but it's reasonable to assume that noone relies on the particular format here.
Signed-off-by Andi Kleen <[email protected]>--- fs/dcache.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)Index:
linux-2.6.31-rc1-ak/fs/dcache.c
===================================================================fs/dcache.c +12
-2@@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,spin_lock (&vfsmount_lock);prepend (&end, &buflen,"\0" ,1 ); -if (d_unlinked (dentry) && - (prepend (&end, &buflen," (deleted)" ,10 ) !=0 )) +if (d_unlinked (dentry)) { +char buf[40 ]; +int n; +if (dentry->d_inode) {/* check needed? */ + n =snprintf (buf,sizeof buf," (deleted; size %lluK)" , + dentry->d_inode->i_size >>10 ); + }else { +strcpy (buf," (deleted)" ); + n =10 ; + } +if (prepend (&end, &buflen, buf, n) !=0 ) goto Elong; + }if (buflen <1 ) goto Elong;Signature · 2 lines
-- [email protected] -- Speaking for myself only.
Signature · 1 line
BRGDS. Alexey Vlasov.
2009/7/13 Alexey Vlasov <[email protected]>:
Quoted 20 lines · Andi Kleen
Quoted 8 lines
Quoted 3 lines · Alexey Vlasov
I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.
You can check for that by doing ls -l
/proc/*/fd| grep deletedIt won't tell you the file size though.
There comes a big list, like this:
lowx------ 1 root root 64 Jul 13 14:58 1763 ->
/tmp/ZCUDXqiMTp(deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 ->/tmp/ZCUDXqiMTp(deleted) lrwx------ 1 root root 64 Jul 13 14:58 1763 ->/tmp/ZCUDXqiMTp(deleted) [cut]
lrwx------ 1 root root 64 Jul 13 15:29 317 ->/tmp/ZCUDXiOHCk(deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 ->/tmp/ZCUDXiOHCk(deleted) lrwx------ 1 root root 64 Jul 13 15:29 317 ->/tmp/ZCUDXiOHCk(deleted)
Quoted 4 lines
But you can see the file date, these are new files and by next search you won't see them any more.
The most interesting is that the /tmp size constantly grows.
Try:
lsof -n | grep deleted
This will show the process holding them more easily, and the file size (in the seventh column). It won't show the date they were opened though.
Quoted 9 lines · Bryan Donlan
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines · Alexey Vlasov
I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.
You can check for that by doing ls -l
/proc/*/fd| grep deletedIt won't tell you the file size though.
Quoted 7 lines
Quoted 5 lines
There comes a big list, like this:
lowx------ 1 root root 64 Jul 13 14:58 1763 ->
/tmp/ZCUDXqiMTp(deleted)The most interesting is that the /tmp size constantly grows.
lsof -n | grep deleted
# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'mysqld 22905 mysql 6u REG
0,13 20 663938322 /tmp/ib1oZTSZ (deleted)
mysqld 22905 mysql 8u REG
0,13 84773 663938424 /tmp/ibRcarbD (deleted)
# df -h /tmp Filesystem Size Used Avail Use% Mounted on tmpfs 4.1G 3.4G 630M 85% /tmp
It will soon be up to 100% full and I'll have to remount.
Signature · 1 line
BRGDS. Alexey Vlasov.
2009/7/13 Alexey Vlasov <[email protected]>:
Quoted 25 lines · Bryan Donlan
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines · Alexey Vlasov
I know that df, du can show different values if some files were deleted, but the processes which opened these files are still acting.
You can check for that by doing ls -l
/proc/*/fd| grep deletedIt won't tell you the file size though.
Quoted 7 lines
Quoted 5 lines
There comes a big list, like this:
lowx------ 1 root root 64 Jul 13 14:58 1763 ->
/tmp/ZCUDXqiMTp(deleted)The most interesting is that the /tmp size constantly grows.
lsof -n | grep deleted
# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'mysqld 22905 mysql 6u REG 0,13 20 663938322
/tmp/ib1oZTSZ(deleted) mysqld 22905 mysql 8u REG 0,13 84773 663938424/tmp/ibRcarbD(deleted)
Quoted 5 lines
# df -h /tmp Filesystem Size Used Avail Use% Mounted on tmpfs 4.1G 3.4G 630M 85% /tmpIt will soon be up to 100% full and I'll have to remount.
Are those the only two results? That should only be adding up to ~84kb of space...
Quoted 8 lines · Bryan Donlan
2009/7/13 Alexey Vlasov <[email protected]>:
Quoted 7 lines
# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'mysqld 22905 mysql 6u REG 0,13 20 663938322
/tmp/ib1oZTSZ(deleted) mysqld 22905 mysql 8u REG 0,13 84773 663938424/tmp/ibRcarbD(deleted)
Quoted 6 lines
Quoted 3 lines
# df -h /tmp Filesystem Size Used Avail Use% Mounted on tmpfs 4.1G 3.4G 630M 85% /tmpAre those the only two results? That should only be adding up to ~84kb of space...
Yes, only two lines where the 7-th column isn't equal to 0
Signature · 1 line
BRGDS. Alexey Vlasov.