difference in space used
Hi all,
Very new to LVM, running into the following problem: We have a logical volume /dev/mapper/vg0-lv2 mounted on /log. While the disk usage for the /log shows only about 23 megs, running df for the logical volume is showing over 1 gig.
The logical volume was previously set to 6 gigs, and when the /log mount point only had 123 megs, the LV was full.
We redirected the output from /log to another location, unmounted /dev/mapper/vg0-lv2, ran fsck, extended the LV to 12 gigs and remounted.
But now the system is filling up again quickly as shown below.
[root@newbox1 ~]# df -T
Filesystem Type 1K-blocks Used Available Use% Mounted on/dev/sda1 ext3 12389324 7195644 4564336 62% /
none tmpfs 1037404 0 1037404 0% /dev/shm/dev/mapper/vg0-lv0
ext3 15118728 1890900 12459828 14% /var/lib
/dev/mapper/vg0-lv1
ext3 5039616 728048 4055568 16% /home
10.202.2.100:/share1
nfs 352982176 125698112 226931088 36% /snap
/dev/mapper/vg0-lv4
ext3 71523868 25955308 41935408 39% /backup
/dev/mapper/vg0-lv2
ext3 12385456 1411556 10470584 12% /log[root@newbox1 ~]# df /log/
Filesystem 1K-blocks Used Available Use% Mounted on/dev/mapper/vg0-lv2 12385456 1391740 10490400 12% /log
[root@newbox1 ~]# du /log/
16 /log/lost+found 23696 /log/httpd 23716 /log/
[root@newbox1 ~]# vgdisplay
--- Volume group ---
VG Name vg0
System ID
Format lvm2 Metadata Areas 10 Metadata Sequence No 18 VG Access read/write VG Status resizable MAX LV 0 Cur LV 4 Open LV 4 Max PV 0 Cur PV 10 Act PV 10 VG Size 107.51 GB PE Size 4.00 MB Total PE 27522 Alloc PE / Size 25812 / 100.83 GB Free PE / Size 1710 / 6.68 GB VG UUID 2WzghL-C7Nf-17oy-ISA9-hZu4-Bhtc-WvHF8Z
[root@newbox1 ~]# lvdisplay /dev/vg0/lv2
--- Logical volume ---
LV Name /dev/vg0/lv2 VG Name vg0 LV UUID DHb2dZ-zYpx-bipG-rjmL-CxBu-q0At-1oOBJB LV Write Access read/write LV Status available # open 1 LV Size 12.00 GB Current LE 3072 Segments 2 Allocation inherit Read ahead sectors 0 Block device 253:2
Any ideas as to what could be going on?
Thank you,
Greg
Signature · 8 lines
Greg Hanson 1506 Gilbert Ave Coeur d'Alene, ID 83815 email [email protected] Phone 208-667-2442 Toll Free 800-949-1889 Fax 775-256-2231
Quoted 6 lines · Greg Hanson
[root@newbox1 ~]# du /log/
16 /log/lost+found 23696 /log/httpd 23716 /log/Any ideas as to what could be going on?
An unlinked file isn't actually deleted until all file handles to it are closed. Since this is your log partition, you probably have broken log rotation script. I bet its moving the files to a new name, but the logging process isn't closing and reopening the logfile.
These files that are unlinked but still have open handles to them continue to consume their usual amount of disk space. FWIW, I have seen this issue particularly noticably with MySQL when the query log is enabled, but that's a wild guess...
Signature · 8 lines
Ross Vandegrift [email protected] "The good Christian should beware of mathematicians, and all those who make empty prophecies. The danger already exists that the mathematicians have made a covenant with the devil to darken the spirit and to confine man in the bonds of Hell." --St. Augustine, De Genesi ad Litteram, Book II, xviii, 37
Quoted 15 lines · Ross Vandegrift
Quoted 7 lines · Greg Hanson
[root@newbox1 ~]# du /log/
16 /log/lost+found 23696 /log/httpd 23716 /log/Any ideas as to what could be going on?
An unlinked file isn't actually deleted until all file handles to it are closed. Since this is your log partition, you probably have broken log rotation script. I bet its moving the files to a new name, but the logging process isn't closing and reopening the logfile.
Quoted 6 lines
These files that are unlinked but still have open handles to them continue to consume their usual amount of disk space. FWIW, I have seen this issue particularly noticably with MySQL when the query log is enabled, but that's a wild guess...
Thanks for the input Ross. You are exactly on target. I found another post on another forum with regards to this,
http://lists.debian.org/debian-isp/2002/05/msg00046.html
and started examining the log rotate scripts. There was an error in the path in the HUP line to restart apache, and there were many deleted log files laying around.
this command
lsof | grep deleted
confirmed that there were several occurrences of error and access logs hanging in there. Did an apachectl stop and start and things went back to normal.
Thank you so much for the response!
Greg
Signature · 11 lines
Greg Hanson Interchange Consulting Perusion 1506 E Gilbert Ave Coeur d'Alene, ID 83815 Email [email protected] Phone 208-667-2442 Toll Free 800-949-1889 Fax 775-256-2231 Web http://www.perusion.com