Recursive chmod/chown OOM kills box with 32MB RAM
I have an old PII server box with just 32MB of RAM.
Yesterday I was preparing it for use by other people, not just me, and went on checking and tightening filesystem permissions with chmod -R and/or chown -R.
On deep directories box started to OOM-kill processes en masse!
I updated kernel to 2.6.15.1 - doesn't help.
I stopped some of more memory hungry processes before running chmod -R - doesn't help.
Details:
2006-01-28_13:35:57.55341 kern.notice: Linux version 2.6.15.1 (root@firebird) (gcc version 3.4.1) #1 Sat Jan 28 14:01:51 EET 2006 2006-01-28_13:35:57.55380 kern.info: BIOS-provided physical RAM map:
2006-01-28_13:35:57.55391 kern.warn: BIOS-e820: 0000000000000000 - 000000000009f800 (usable) 2006-01-28_13:35:57.55402 kern.warn: BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved) 2006-01-28_13:35:57.55413 kern.warn: BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved) 2006-01-28_13:35:57.55423 kern.warn: BIOS-e820: 0000000000100000 - 0000000002000000 (usable) 2006-01-28_13:35:57.55434 kern.warn: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) 2006-01-28_13:35:57.55444 kern.warn: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) 2006-01-28_13:35:57.55455 kern.warn: BIOS-e820: 00000000ffe80000 - 0000000100000000 (reserved) 2006-01-28_13:35:57.55466 kern.notice: 32MB LOWMEM available. 2006-01-28_13:35:57.55474 kern.debug: On node 0 totalpages: 8192 2006-01-28_13:35:57.55483 kern.debug: DMA zone: 4096 pages, LIFO batch:0 2006-01-28_13:35:57.55492 kern.debug: DMA32 zone: 0 pages, LIFO batch:0 2006-01-28_13:35:57.55502 kern.debug: Normal zone: 4096 pages, LIFO batch:0 2006-01-28_13:35:57.55511 kern.debug: HighMem zone: 0 pages, LIFO batch:0
Here is the first OOM kill in the last test run:
13:36:00.85 ip_conntrack version 2.4 (256 buckets, 2048 max) - 216 bytes per conntrack
13:36:04.10 NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
13:36:04.10 NFSD: recovery directory /var/lib/nfs/v4recovery doesn't exist
13:36:04.10 NFSD: starting 90-second grace period
13:39:27.71 SysRq : HELP : loglevel0-8 reBoot tErm Full kIll saK showMem Nice powerOff showPc unRaw Sync showTasks Unmount 13:39:28.21 SysRq : HELP : loglevel0-8 reBoot tErm Full kIll saK showMem Nice powerOff showPc unRaw Sync showTasks Unmount 13:40:26.63 SysRq : Changing Loglevel
13:40:26.64 Loglevel set to 9
13:41:07.46 oom-killer: gfp_mask=0x200d2, order=0
13:41:07.47 Mem-info:
13:41:07.47 DMA per-cpu:
13:41:07.47 cpu 0 hot: low 0, high 0, batch 1 used:0
13:41:07.47 cpu 0 cold: low 0, high 0, batch 1 used:0
13:41:07.47 DMA32 per-cpu: empty
13:41:07.47 Normal per-cpu:
13:41:07.48 cpu 0 hot: low 0, high 0, batch 1 used:0
13:41:07.48 cpu 0 cold: low 0, high 0, batch 1 used:0
13:41:07.48 HighMem per-cpu: empty
13:41:07.48 Free pages: 952kB (0kB HighMem)
13:41:07.48 Active:2217 inactive:2002 dirty:0 writeback:4 unstable:0 free:238 slab:1383 mapped:23 pagetables:243 13:41:07.48 DMA free:432kB min:360kB low:448kB high:540kB active:4556kB inactive:3988kB present:16384kB pages_scanned:390 all_unreclaimable? no 13:41:07.48 lowmem_reserve[]: 0 0 16 16
13:41:07.48 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no 13:41:07.48 lowmem_reserve[]: 0 0 16 16
13:41:07.48 Normal free:520kB min:360kB low:448kB high:540kB active:4312kB inactive:4020kB present:16384kB pages_scanned:4 all_unreclaimable? no 13:41:07.48 lowmem_reserve[]: 0 0 0 0
13:41:07.48 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no 13:41:07.48 lowmem_reserve[]: 0 0 0 0
13:41:07.48 DMA: 2*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 432kB
13:41:07.48 DMA32: empty
13:41:07.48 Normal: 40*4kB 1*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 520kB 13:41:07.48 HighMem: empty
13:41:07.48 Swap cache: add 6654, delete 6608, find 1126/2081, race 0+0
13:41:07.49 Free swap = 92080kB
13:41:07.49 Total swap = 98296kB
13:41:07.49 Free swap: 92080kB
13:41:07.49 8192 pages of RAM
13:41:07.49 0 pages of HIGHMEM
13:41:07.49 1301 reserved pages
13:41:07.49 3949 pages shared
13:41:07.49 46 pages swap cached
13:41:07.49 0 pages dirty
13:41:07.49 4 pages writeback
13:41:07.49 23 pages mapped
13:41:07.49 1383 pages slab
13:41:07.49 243 pages pagetables
13:41:07.49 Out of Memory: Killed process 1173 (top).
Process tree on this box looks like this while I was doing tests (PIDs won't match, I rebooted the box):
PID TTY STAT TIME COMMAND
1 ? S 0:00 /bin/sh /init HOME=/ TERM=linux devfs=nomount
2 ? SWN 0:00 [ksoftirqd/0]
3 ? SW 0:00 [watchdog/0]
4 ? SW< 0:00 [events/0]
5 ? SW< 0:00 [khelper]
6 ? SW< 0:00 [kthread]
8 ? SW< 0:00 [kblockd/0]
164 ? SW< 0:00 [aio/0]
239 ? SW< 0:00 [kseriod]
349 ? SW< 0:00 [scsi_eh_0]
387 ? SW< 0:00 [ata/0]
648 ? SW< 0:00 [reiserfs/0]
940 ? SW 0:00 [pdflush]
943 ? SW 0:00 [pdflush]
1518 ? SW< 0:00 [nfsd4]
1533 ? SW< 0:00 [rpciod/0]
163 ? SW 0:00 [kswapd0]
559 ? S< 0:00 udevd
784 ? S 0:00 rpc.portmap
1088 ? S 0:00 inetd
1118 ? S 0:00 svscan /var/service PATH=/bin:/usr/bin
1122 ? S 0:00 supervise fw PATH=/bin:/usr/bin
1123 ? S 0:00 supervise sshd PATH=/bin:/usr/bin
1127 ? S 0:00 sshd -D -e -p22 -u0
1643 ? S 0:00 sshd -D -e -p22 -u0
1644 pts/0 S 0:00 -bash USER=vda LOGNAME=vda HOME=/home/vda PATH=/usr/bin:/bin:/usr/sbin:/sbin MAIL=/var/mail/vda SHELL=/bin/bash SSH_CLIENT=172.17.2.38 33504 22 SSH_TTY=/dev/pts/0 TERM=xterm
1653 pts/0 S 0:05 mc MANPATH=/usr/man HOSTNAME=pegasus.port.imtp.ilyichevsk.odessa.ua TERM=xterm SHELL=/bin/bash SSH_CLIENT=172.17.2.38 33504 22 SSH_TTY=/dev/pts/0 USER=vda LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=
1836 pts/0 S 0:00 /bin/bash -c psahe >psahe MANPATH=/usr/man HOSTNAME=pegasus.port.imtp.ilyichevsk.odessa.ua TERM=xterm SHELL=/bin/bash SSH_CLIENT=172.17.2.38 33504 22 SSH_TTY=/dev/pts/0 USER=vda LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.jpg=01;35:*.jpeg
1837 pts/0 S 0:00 /bin/sh /home/vda/bin/psahe MANPATH=/usr/man HOSTNAME=pegasus.port.imtp.ilyichevsk.odessa.ua SHELL=/bin/bash TERM=xterm SSH_CLIENT=172.17.2.38 33504 22 SSH_TTY=/dev/pts/0 USER=vda LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.jpg=01;35:*.
1838 pts/0 R 0:00 ps -AH e --width=500 MANPATH=/usr/man HOSTNAME=pegasus.port.imtp.ilyichevsk.odessa.ua TERM=xterm SHELL=/bin/bash SSH_CLIENT=172.17.2.38 33504 22 OLDPWD=/.local/tmp SSH_TTY=/dev/pts/0 USER=vda LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:
1839 pts/0 D 0:00 most MANPATH=/usr/man HOSTNAME=pegasus.port.imtp.ilyichevsk.odessa.ua TERM=xterm SHELL=/bin/bash SSH_CLIENT=172.17.2.38 33504 22 OLDPWD=/.local/tmp SSH_TTY=/dev/pts/0 USER=vda LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.jpg=01;35:*.jp
1124 ? S 0:00 supervise ntp PATH=/bin:/usr/bin
1125 ? S 0:00 supervise log PATH=/bin:/usr/bin
1129 ? S 0:00 multilog t n5 /var/log/service/ntp
1131 ? S 0:00 supervise dhcp PATH=/bin:/usr/bin
1132 ? S 0:00 supervise log PATH=/bin:/usr/bin
1144 ? S 0:00 multilog t /var/log/service/dhcp
1145 ? S 0:00 supervise watcher PATH=/bin:/usr/bin
1152 ? S 0:00 /bin/sh ./run PATH=/bin:/usr/bin
1823 ? S 0:00 sleep 33
1150 ? S 0:00 supervise klog PATH=/bin:/usr/bin
1167 ? S 0:00 socklog ucspi
1151 ? S 0:00 supervise log PATH=/bin:/usr/bin
1158 ? S 0:00 svlogd -tt /var/log/service/klog
1155 ? S 0:00 supervise syslog PATH=/bin:/usr/bin
1168 ? S 0:00 socklog unix /dev/log PATH=/bin:/usr/bin PWD=/.local/var/service/syslog SHLVL=0 GID=50 UID=59
1156 ? S 0:00 supervise log PATH=/bin:/usr/bin
1174 ? S 0:00 svlogd /var/log/service/syslog
1161 ? S 0:00 supervise pop3 PATH=/bin:/usr/bin
1183 ? S 0:00 tcpserver -v -R -H -l 0 -c 40 0.0.0.0 pop3 setuidgid root qmail-popup checkpassword qmail-pop3d maildir
1162 ? S 0:00 supervise log PATH=/bin:/usr/bin
1199 ? S 0:00 multilog t /var/log/service/pop3
1171 ? S 0:00 supervise top PATH=/bin:/usr/bin
1200 ? S 0:14 top c s TERM=linux
1172 ? S 0:00 supervise mysql PATH=/bin:/usr/bin
1173 ? S 0:00 supervise log PATH=/bin:/usr/bin
1190 ? S 0:00 multilog t n5 /var/log/service/mysql
1198 ? S 0:00 supervise getty_tty2 PATH=/bin:/usr/bin
1598 tty2 S 0:00 agetty 38400 /dev/tty2 linux TERM=linux
1213 ? S 0:00 supervise getty_tty3 PATH=/bin:/usr/bin
1619 tty3 S 0:00 agetty 38400 /dev/tty3 linux TERM=linux
1218 ? S 0:00 supervise getty_tty4 PATH=/bin:/usr/bin
1224 tty4 S 0:00 agetty 38400 /dev/tty4 linux TERM=linux
1225 ? S 0:00 supervise getty_tty5 PATH=/bin:/usr/bin
1244 tty5 S 0:00 agetty 38400 /dev/tty5 linux TERM=linux
1235 ? S 0:00 supervise getty_tty6 PATH=/bin:/usr/bin
1243 tty6 S 0:00 agetty 38400 /dev/tty6 linux TERM=linux
1242 ? S 0:00 supervise getty_tty7 PATH=/bin:/usr/bin
1252 tty7 S 0:00 agetty 38400 /dev/tty7 linux TERM=linux
1253 ? S 0:00 supervise getty_tty8 PATH=/bin:/usr/bin
1260 tty8 S 0:00 agetty 38400 /dev/tty8 linux TERM=linux
1261 ? S 0:00 supervise getty_tty1 PATH=/bin:/usr/bin
1265 tty1 S 0:00 agetty 38400 /dev/tty1 linux TERM=linux
1274 ? S 0:00 supervise httpd PATH=/bin:/usr/bin
1276 ? S 0:00 tcpserver -v -R -H -l 0 -c 40 0.0.0.0 www setuidgid root httpd -X -f /.local/var/service/httpd/httpd.conf PATH=/bin:/usr/bin
1275 ? S 0:00 supervise log PATH=/bin:/usr/bin
1277 ? S 0:00 multilog t /var/log/service/httpd
1278 ? S 0:00 supervise httpd_ssl PATH=/bin:/usr/bin
1280 ? S 0:00 stunnel -d 443 -D 6 -p https.pem -S 0 -f -P none -N httpd -l setuidgid -- setuidgid root httpd -X -f /.local/var/service/httpd_ssl/httpd.conf
1279 ? S 0:00 supervise log PATH=/bin:/usr/bin
1332 ? S 0:00 multilog /var/log/service/httpd_ssl
1281 ? S 0:00 supervise qmail PATH=/bin:/usr/bin
1282 ? S 0:00 supervise log PATH=/bin:/usr/bin
1303 ? S 0:00 multilog t s300000 /var/log/service/qmail -@* status: local * remote * -@* new msg * -@* end msg * /var/log/service/qmail_nostatus -@* delivery *: success: * /var/log/service/qmail_nosuccess -@* info msg *: * -@* starting delivery *: * /var/log/service/qmail_problems
1296 ? S 0:00 supervise smtp PATH=/bin:/usr/bin
1299 ? S 0:00 tcpserver -v -R -H -l 0 -c 60 0.0.0.0 25 setuidgid mail smtpfront-qmail QMAILQUEUE=/usr/bin/qmail-scanner-queue.pl CVM_SASL_PLAIN=/usr/app/cvm-0.11/bin/cvm-unix
1297 ? S 0:00 supervise log PATH=/bin:/usr/bin
1304 ? S 0:00 multilog t s200000 /var/log/service/smtp
1310 ? S 0:00 supervise smb_n PATH=/bin:/usr/bin
1311 ? S 0:00 supervise log PATH=/bin:/usr/bin
1348 ? S 0:00 multilog t /var/log/service/smb_n
1320 ? S 0:00 supervise smb_s PATH=/bin:/usr/bin
1321 ? S 0:00 supervise log PATH=/bin:/usr/bin
1365 ? S 0:00 multilog t /var/log/service/smb_s
1322 ? S 0:00 supervise smb_w PATH=/bin:/usr/bin
1323 ? S 0:00 supervise log PATH=/bin:/usr/bin
1363 ? S 0:00 multilog t /var/log/service/smb_w
1346 ? S 0:00 supervise nfs PATH=/bin:/usr/bin
1364 ? S 0:00 /bin/sh ./run PATH=/bin:/usr/bin
1542 ? S 0:00 sleep 32000
1380 ? S 0:00 supervise proxy-tcp PATH=/bin:/usr/bin
1391 ? S 0:00 tcpserver -U -v -R -H -l 0 -c 100 0.0.0.0 9123 ./startproxy GID=50 UID=50
1381 ? S 0:00 supervise log PATH=/bin:/usr/bin
1404 ? S 0:00 multilog t /var/log/service/proxy-tcp
1392 ? S 0:00 supervise pgsql PATH=/bin:/usr/bin
1393 ? S 0:00 supervise log PATH=/bin:/usr/bin
1414 ? S 0:00 multilog t /var/log/service/pgsql
1396 ? S 0:00 supervise once PATH=/bin:/usr/bin
1405 ? S 0:00 supervise ovpn-1 PATH=/bin:/usr/bin
1406 ? S 0:00 supervise log PATH=/bin:/usr/bin
1445 ? S 0:00 multilog /var/log/service/ovpn-1
1443 ? S 0:00 supervise automount PATH=/bin:/usr/bin
1447 ? S 0:00 automount -f -s -v --timeout 15 /.local/mnt/auto program /root/bin/mapper.sh
1444 ? S 0:00 supervise log PATH=/bin:/usr/bin
1452 ? S 0:00 multilog t n5 /var/log/service/automount
1121 ? S 0:00 sleep 32000
1524 ? SW 0:00 [nfsd]
1525 ? SW 0:00 [nfsd]
1526 ? SW 0:00 [nfsd]
1527 ? SW 0:00 [nfsd]
1532 ? SW 0:00 [lockd]
1534 ? S 0:00 rpc.mountd
1536 ? S 0:00 rpc.statddaemontools, bash and some other programs are compiled againts dietlibc or uclibc. top b n1 output:
16:01:33 up 15 min, 2 users, load average: 0,32, 0,16, 0,17
121 processes: 120 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 3,3% user 2,4% system 0,0% nice 14,1% iowait 80,0% idle Mem: 27564k av, 25088k used, 2476k free, 0k shrd, 936k buff
12944k active, 2468k inactive
Swap: 98296k av, 0k used, 98296k free 9432k cached
(swap usage is 0 here because box was just rebooted. It starts to use swap
in normal use. A few mins later: "98296k av, 152k used, 98144k free")
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
1870 root 17 0 2164 1168 804 R 8,1 4,2 0:00 0 top
1 root 15 0 1044 616 516 S 0,0 2,2 0:00 0 init
2 root 34 19 0 0 0 SWN 0,0 0,0 0:00 0 ksoftirqd/0
3 root RT 0 0 0 0 SW 0,0 0,0 0:00 0 watchdog/0
4 root 10 -5 0 0 0 SW< 0,0 0,0 0:00 0 events/0
5 root 11 -5 0 0 0 SW< 0,0 0,0 0:00 0 khelper
6 root 11 -5 0 0 0 SW< 0,0 0,0 0:00 0 kthread
8 root 10 -5 0 0 0 SW< 0,0 0,0 0:00 0 kblockd/0
164 root 17 -5 0 0 0 SW< 0,0 0,0 0:00 0 aio/0
163 root 15 0 0 0 0 SW 0,0 0,0 0:00 0 kswapd0
239 root 10 -5 0 0 0 SW< 0,0 0,0 0:00 0 kseriod
349 root 11 -5 0 0 0 SW< 0,0 0,0 0:00 0 scsi_eh_0
387 root 11 -5 0 0 0 SW< 0,0 0,0 0:00 0 ata/0
559 root 16 -4 128 16 4 S < 0,0 0,0 0:00 0 udevd
648 root 10 -5 0 0 0 SW< 0,0 0,0 0:00 0 reiserfs/0
784 rpc 16 0 1444 560 460 S 0,0 2,0 0:00 0 rpc.portmap
940 root 15 0 0 0 0 SW 0,0 0,0 0:00 0 pdflush
943 root 15 0 0 0 0 SW 0,0 0,0 0:00 0 pdflush
1088 root 18 0 1344 460 384 S 0,0 1,6 0:00 0 inetd
1118 root 16 0 132 24 8 S 0,0 0,0 0:00 0 svscan
1121 root 18 0 1816 464 388 S 0,0 1,6 0:00 0 sleep
1122 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1123 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1124 root 18 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1125 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1127 root 16 0 2568 1192 940 S 0,0 4,3 0:01 0 sshd
1129 daemon 18 0 108 28 16 S 0,0 0,1 0:00 0 multilog
1131 root 18 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1132 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1144 daemon 17 0 112 32 16 S 0,0 0,1 0:00 0 multilog
1145 root 16 0 100 20 8 S 0,0 0,0 0:00 0 supervise
1150 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1151 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1152 root 16 0 1048 624 524 S 0,0 2,2 0:00 0 run
1155 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1156 root 16 0 100 20 8 S 0,0 0,0 0:00 0 supervise
1158 logger 18 0 120 40 20 S 0,0 0,1 0:00 0 svlogd
1161 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1162 root 16 0 104 16 8 S 0,0 0,0 0:00 0 supervise
1167 root 16 0 116 32 12 S 0,0 0,1 0:00 0 socklog
1168 logger 16 0 120 32 12 S 0,0 0,1 0:00 0 socklog
1171 root 16 0 104 16 8 S 0,0 0,0 0:00 0 supervise
1172 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1173 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1174 logger 18 0 116 40 20 S 0,0 0,1 0:00 0 svlogd
1183 root 17 0 128 48 32 S 0,0 0,1 0:00 0 tcpserver
1190 daemon 17 0 108 28 16 S 0,0 0,1 0:00 0 multilog
1198 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1199 mail 15 0 108 28 16 S 0,0 0,1 0:00 0 multilog
1200 user0 17 0 1840 1004 728 S 0,0 3,6 0:14 0 top
1213 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1218 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1224 root 16 0 1312 456 380 S 0,0 1,6 0:00 0 agetty
1225 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1235 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1242 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1243 root 16 0 1308 452 380 S 0,0 1,6 0:00 0 agetty
1244 root 16 0 1312 456 380 S 0,0 1,6 0:00 0 agetty
1252 root 16 0 1312 456 380 S 0,0 1,6 0:00 0 agetty
1253 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1260 root 16 0 1308 452 380 S 0,0 1,6 0:00 0 agetty
1261 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1265 root 17 0 1308 452 380 S 0,0 1,6 0:00 0 agetty
1274 root 16 0 100 20 8 S 0,0 0,0 0:00 0 supervise
1275 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1276 root 15 0 124 40 32 S 0,0 0,1 0:00 0 tcpserver
1277 apache 17 0 112 32 16 S 0,0 0,1 0:00 0 multilog
1278 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1279 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1280 root 16 0 2888 1340 1072 S 0,0 4,8 0:00 0 stunnel
1281 root 18 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1282 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1296 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1297 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1299 root 18 0 128 40 28 S 0,0 0,1 0:00 0 tcpserver
1303 mail 17 0 116 36 16 S 0,0 0,1 0:00 0 multilog
1304 mail 17 0 112 36 16 S 0,0 0,1 0:00 0 multilog
1310 root 17 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1311 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1320 root 17 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1321 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1322 root 17 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1323 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1332 apache 17 0 112 36 16 S 0,0 0,1 0:00 0 multilog
1346 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1348 logger 17 0 112 32 16 S 0,0 0,1 0:00 0 multilog
1363 logger 17 0 112 32 16 S 0,0 0,1 0:00 0 multilog
1364 root 16 0 1056 628 520 S 0,0 2,2 0:00 0 run
1365 logger 17 0 112 36 16 S 0,0 0,1 0:00 0 multilog
1380 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1381 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1391 daemon 18 0 124 36 28 S 0,0 0,1 0:00 0 tcpserver
1392 root 18 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1393 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1396 root 17 0 104 24 8 S 0,0 0,0 0:00 0 supervise
1404 daemon 16 0 108 28 16 S 0,0 0,1 0:00 0 multilog
1405 root 18 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1406 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1414 mysql 15 0 112 36 16 S 0,0 0,1 0:00 0 multilog
1443 root 16 0 100 16 8 S 0,0 0,0 0:00 0 supervise
1444 root 16 0 104 20 8 S 0,0 0,0 0:00 0 supervise
1445 daemon 15 0 112 32 16 S 0,0 0,1 0:00 0 multilog
1447 root 16 0 1440 648 540 S 0,0 2,3 0:00 0 automount
1452 logger 16 0 112 32 16 S 0,0 0,1 0:00 0 multilog
1518 root 10 -5 0 0 0 SW< 0,0 0,0 0:00 0 nfsd4
1524 root 15 0 0 0 0 SW 0,0 0,0 0:00 0 nfsd
1525 root 15 0 0 0 0 SW 0,0 0,0 0:00 0 nfsd
1526 root 15 0 0 0 0 SW 0,0 0,0 0:00 0 nfsd
1527 root 15 0 0 0 0 SW 0,0 0,0 0:00 0 nfsd
1532 root 16 0 0 0 0 SW 0,0 0,0 0:00 0 lockd
1533 root 11 -5 0 0 0 SW< 0,0 0,0 0:00 0 rpciod/0
1534 root 17 0 1512 584 408 S 0,0 2,1 0:00 0 rpc.mountd
1536 root 16 0 3380 1392 952 S 0,0 5,0 0:00 0 rpc.statd
1542 root 18 0 1816 464 388 S 0,0 1,6 0:00 0 sleep
1598 root 16 0 1312 456 380 S 0,0 1,6 0:00 0 agetty
1619 root 16 0 1308 452 380 S 0,0 1,6 0:00 0 agetty
1644 root 16 0 1152 824 596 S 0,0 2,9 0:00 0 bash
1653 root 15 0 1412 948 592 S 0,0 3,4 0:05 0 mc
1854 root 18 0 1816 464 388 S 0,0 1,6 0:00 0 sleep
1869 root 18 0 1052 572 468 S 0,0 2,0 0:00 0 bash--
vda
Quoted 11 lines · Denis Vlasenko
I have an old PII server box with just 32MB of RAM.
Yesterday I was preparing it for use by other people, not just me, and went on checking and tightening filesystem permissions with chmod -R and/or chown -R.
On deep directories box started to OOM-kill processes en masse!
I updated kernel to 2.6.15.1 - doesn't help.
I stopped some of more memory hungry processes before running chmod -R - doesn't help.
More details which might be relevant.
I did not alter anything in /proc/sys/vm, maybe I should?
# cd /proc/sys/vm # for a in *; do echo "$a: `cat "$a"`"; done block_dump: 0 dirty_background_ratio: 10 dirty_expire_centisecs: 3000 dirty_ratio: 40 dirty_writeback_centisecs: 500 hugetlb_shm_group: 0 laptop_mode: 0 legacy_va_layout: 0 lowmem_reserve_ratio: 256 256 32 max_map_count: 65536 min_free_kbytes: 724 nr_hugepages: 0 nr_pdflush_threads: 2 overcommit_memory: 0 overcommit_ratio: 50 page-cluster: 3 swap_token_timeout: 300 swappiness: 60 vfs_cache_pressure: 100
SCSI controller and disks attached to it:
lspci says: "00:0b.0 SCSI storage controller: Adaptec AIC-7880U (rev 01)" boot log:
13:48:34.50 ahc_pci:0:11:0: Using left over BIOS settings 13:48:34.50 scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
13:48:34.50 <Adaptec aic7880 Ultra SCSI adapter>
13:48:34.50 aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
13:48:34.50
13:48:34.50 Vendor: IBM Model: DNES-309170W Rev: SAH0 13:48:34.50 Type: Direct-Access ANSI SCSI revision: 03 13:48:34.50 scsi0:A:0:0: Tagged Queuing enabled. Depth 8 13:48:34.50 target0:0:0: Beginning Domain Validation 13:48:34.50 target0:0:0: wide asynchronous. 13:48:34.50 target0:0:0: FAST-20 WIDE SCSI 40.0 MB/s ST (50 ns, offset 8) 13:48:34.50 target0:0:0: Domain Validation skipping write tests 13:48:34.50 target0:0:0: Ending Domain Validation 13:48:34.50 Vendor: IBM Model: DNES-309170W Rev: SAH0 13:48:34.50 Type: Direct-Access ANSI SCSI revision: 03 13:48:34.50 scsi0:A:1:0: Tagged Queuing enabled. Depth 8 13:48:34.50 target0:0:1: Beginning Domain Validation 13:48:34.50 target0:0:1: wide asynchronous. 13:48:34.50 target0:0:1: FAST-20 WIDE SCSI 40.0 MB/s ST (50 ns, offset 8) 13:48:34.50 target0:0:1: Domain Validation skipping write tests 13:48:34.50 target0:0:1: Ending Domain Validation 13:48:34.50 Vendor: SEAGATE Model: ST39140N Rev: 1498 13:48:34.50 Type: Direct-Access ANSI SCSI revision: 02 13:48:34.50 scsi0:A:3:0: Tagged Queuing enabled. Depth 8 13:48:34.50 target0:0:3: Beginning Domain Validation 13:48:34.50 target0:0:3: FAST-20 SCSI 20.0 MB/s ST (50 ns, offset 15) 13:48:34.50 target0:0:3: Domain Validation skipping write tests 13:48:34.50 target0:0:3: Ending Domain Validation
13:48:34.50 libata version 1.20 loaded.
13:48:34.50 SCSI device sda: 17916240 512-byte hdwr sectors (9173 MB) 13:48:34.50 SCSI device sda: drive cache: write back 13:48:34.50 SCSI device sda: 17916240 512-byte hdwr sectors (9173 MB) 13:48:34.50 SCSI device sda: drive cache: write back 13:48:34.50 sda: sda1 sda2
13:48:34.50 sd 0:0:0:0: Attached scsi disk sda
13:48:34.51 SCSI device sdb: 17916240 512-byte hdwr sectors (9173 MB) 13:48:34.51 SCSI device sdb: drive cache: write back 13:48:34.51 SCSI device sdb: 17916240 512-byte hdwr sectors (9173 MB) 13:48:34.51 SCSI device sdb: drive cache: write back 13:48:34.51 sdb: sdb1 sdb2
13:48:34.51 sd 0:0:1:0: Attached scsi disk sdb
13:48:34.51 SCSI device sdc: 17783240 512-byte hdwr sectors (9105 MB) 13:48:34.51 SCSI device sdc: drive cache: write back 13:48:34.51 SCSI device sdc: 17783240 512-byte hdwr sectors (9105 MB) 13:48:34.51 SCSI device sdc: drive cache: write back 13:48:34.51 sdc: sdc1 sdc2 sdc3
--
vda
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .
in a top directory of tree containing ~21938 files on reiser3 partition:
/dev/sdc3 on /.3 type reiserfs (rw,noatime)
causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills while chown's RSS stays below 4MB.
--
vda
Quoted 11 lines · would Denis Vlasenko
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
/dev/sdc3on /.3 type reiserfs (rw,noatime)
Quoted 12 lines
causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills while chown's RSS stays below 4MB.
--
vda
Chris, would you like to handle this?
Thanks,
Hans
Quoted 10 lines · Hans Reiser
Quoted 9 lines · would Denis Vlasenko
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
Quoted 8 lines
Quoted 8 lines
/dev/sdc3on /.3 type reiserfs (rw,noatime)causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills while chown's RSS stays below 4MB.
In order for the journaled filesystems to make sure the FS is consistent after a crash, we need to keep some blocks in memory until other blocks have been written. These blocks are pinned, and can't be freed until a certain amount of io is done.
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024 /dev/xxxx
The filesystem must be unmounted first.
-chris
Quoted 29 lines · Chris Mason
Quoted 10 lines · Hans Reiser
Quoted 9 lines · would Denis Vlasenko
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
Quoted 8 lines
Quoted 8 lines
/dev/sdc3on /.3 type reiserfs (rw,noatime)causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills while chown's RSS stays below 4MB.
In order for the journaled filesystems to make sure the FS is consistent after a crash, we need to keep some blocks in memory until other blocks have been written. These blocks are pinned, and can't be freed until a certain amount of io is done.
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
Quoted 6 lines
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxxThe filesystem must be unmounted first.
Will try this and report the result.
Please consider printing a big fat warning at mount time if total RAM on the system is close to sum of RAM space required for all currently mounted reiserfs partitions...
--
vda
Quoted 46 lines · Denis Vlasenko
Quoted 34 lines · Chris Mason
Quoted 14 lines · Hans Reiser
Quoted 11 lines · would Denis Vlasenko
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
/dev/sdc3on /.3 type reiserfs (rw,noatime)Quoted 8 lines
Quoted 8 lines
causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills while chown's RSS stays below 4MB.
In order for the journaled filesystems to make sure the FS is consistent after a crash, we need to keep some blocks in memory until other blocks have been written. These blocks are pinned, and can't be freed until a certain amount of io is done.
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
Quoted 8 lines
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxxThe filesystem must be unmounted first.
Quoted 11 lines
Will try this and report the result.
Please consider printing a big fat warning at mount time if total RAM on the system is close to sum of RAM space required for all currently mounted reiserfs partitions...
--
vda
I already suggested this to Chris.;-) I agree.
Best,
Hans
Quoted 32 lines · Denis Vlasenko
Quoted 30 lines · Chris Mason
Quoted 11 lines · Hans Reiser
Quoted 9 lines · would Denis Vlasenko
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
Quoted 8 lines
Quoted 8 lines
/dev/sdc3on /.3 type reiserfs (rw,noatime)causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills while chown's RSS stays below 4MB.
In order for the journaled filesystems to make sure the FS is consistent after a crash, we need to keep some blocks in memory until other blocks have been written. These blocks are pinned, and can't be freed until a certain amount of io is done.
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxx
Quoted 8 lines
Quoted 1 line
The filesystem must be unmounted first.
Will try this and report the result.
Please consider printing a big fat warning at mount time if total RAM on the system is close to sum of RAM space required for all currently mounted reiserfs partitions...
I would think that rather than warn about the problem that it would be better to teach the filesystem not to use all of memory, and to stop and wait for i/o to finish rather than pin so many pages that the system becomes unusable. For definitions of usable which include not killing random small processes because the filesystem code isn't playing nicely.
Would some global count of reiser_pinned_pages be possible as a way to track the problem?
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 6 lines · Chris Mason
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxxThe filesystem must be unmounted first.
It doesn't want to, for no obvious reason:
# mount rootfs on / type rootfs (rw) /dev/root on / type ext2 (ro,nogrpid) none on /proc type proc (rw,nodiratime) none on /sys type sysfs (rw) none on /dev type ramfs (rw) /dev/sda2 on /.share type ext2 (rw,noatime,nogrpid) /dev/sdb2 on /.1 type reiserfs (rw,noatime) /dev/sdc2 on /.2 type reiserfs (rw,noatime) /dev/sdc3 on /.3 type reiserfs (rw,noatime) /dev/sda2 on /.local type ext2 (rw,noatime,nogrpid) none on /dev/pts type devpts (rw) automount(pid1043) on /.local/mnt/auto type autofs (rw) # umount /.3 umount: /.3: device is busy # mount -o ro,remount /.3 # umount /.3 umount: /.3: device is busy # mount rootfs on / type rootfs (rw) /dev/root on / type ext2 (ro,nogrpid) none on /proc type proc (rw,nodiratime) none on /sys type sysfs (rw) none on /dev type ramfs (rw) /dev/sda2 on /.share type ext2 (rw,noatime,nogrpid) /dev/sdb2 on /.1 type reiserfs (rw,noatime) /dev/sdc2 on /.2 type reiserfs (rw,noatime) /dev/sdc3 on /.3 type reiserfs (ro,noatime) /dev/sda2 on /.local type ext2 (rw,noatime,nogrpid) none on /dev/pts type devpts (rw) automount(pid1043) on /.local/mnt/auto type autofs (rw) # reiserfstune /dev/sdc3 reiserfstune: Reiserfstune is not allowed to be run on mounted filesystem. # lsof -nP | grep -F '/.3' # lsof -nP | grep -F '/dev/sdc'
--
vda
Quoted 24 lines · Denis Vlasenko
Quoted 6 lines · Chris Mason
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxxThe filesystem must be unmounted first.
It doesn't want to, for no obvious reason:
# mount rootfs on / type rootfs (rw) /dev/root on / type ext2 (ro,nogrpid) none on /proc type proc (rw,nodiratime) none on /sys type sysfs (rw) none on /dev type ramfs (rw) /dev/sda2 on /.share type ext2 (rw,noatime,nogrpid) /dev/sdb2 on /.1 type reiserfs (rw,noatime) /dev/sdc2 on /.2 type reiserfs (rw,noatime) /dev/sdc3 on /.3 type reiserfs (rw,noatime) /dev/sda2 on /.local type ext2 (rw,noatime,nogrpid) none on /dev/pts type devpts (rw) automount(pid1043) on /.local/mnt/auto type autofs (rw) # umount /.3
Quoted 8 lines
umount: /.3: device is busy # mount -o ro,remount /.3 # umount /.3 umount: /.3: device is busy # reiserfstune /dev/sdc3 reiserfstune: Reiserfstune is not allowed to be run on mounted filesystem. # lsof -nP | grep -F '/.3' # lsof -nP | grep -F '/dev/sdc'
Removed it from /etc/fstab, after reboot:
# reiserfstune -s 1024 /dev/sdc3 reiserfstune: Journal device has not been specified. Assuming journal is on the main device (/dev/sdc3).
Current parameters:
Filesystem state: consistent
Reiserfs super block in block 16 on 0x823 of format 3.6 with standard journal Count of blocks on the device: 2094474
Number of bitmaps: 64
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 1019710 Root block: 32941
Filesystem is cleanly umounted
Tree height: 4
Hash function used to sort names: "r5"
Objectid map size 54, max 972
Journal parameters:
Device [0x0]
Magic [0x2cac04df]
Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks
Max batch size 900 blocks
Max commit age 30
Blocks reserved by journal: 0
Fs state field: 0x0:sb_version: 2
inode generation number: 293892
UUID: 7b3aa1ab-40bd-44da-98dc-b37b21f7add0
LABEL:
Set flags in SB:
ATTRIBUTES CLEAN
reiserfstune: Current journal parameters:
Device [0x0]
Magic [0x2cac04df]
Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks
Max batch size 900 blocks
Max commit age 30
WARNING: wrong transaction max size (1024). Changed to 511
reiserfstune: New journal parameters:
Device [0x0]
Magic [0x70dbc903]
Size 1024 blocks (including 1 for journal header) (first block 18) Max transaction length 511 blocks
Max batch size 449 blocks
Max commit age 30
Reiserfs super block in block 16 on 0x823 of format 3.6 with non-standard journal Count of blocks on the device: 2094474
Number of bitmaps: 64
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 1019710 Root block: 32941
Filesystem is cleanly umounted
Tree height: 4
Hash function used to sort names: "r5"
Objectid map size 54, max 972
Journal parameters:
Device [0x0]
Magic [0x70dbc903]
Size 1024 blocks (including 1 for journal header) (first block 18) Max transaction length 511 blocks
Max batch size 449 blocks
Max commit age 30
Blocks reserved by journal: 8193
Fs state field: 0x0:sb_version: 2
inode generation number: 293892
UUID: 7b3aa1ab-40bd-44da-98dc-b37b21f7add0
LABEL:
Set flags in SB:
ATTRIBUTES CLEAN
reiserfstune: ATTENTION: YOU ARE ABOUT TO SETUP THE NEW JOURNAL FOR THE "/dev/sdc3"! AREA OF "/dev/sdc3" DEDICATED FOR JOURNAL WILL BE ZEROED!
Continue (y/n):y
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok
# mount /dev/sdc3 /.3 -o noatime mount: you must specify the filesystem type
# dmesg | tail -4 br: topology change detected, propagating br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3.
# reiserfsck /dev/sdc3 reiserfsck 3.6.11 (2003 www.namesys.com)
*************************************************************
** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. **
*************************************************************
Will read-only check consistency of the filesystem on /dev/sdc3 Will put log info to 'stdout'
Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Failed to open the journal device ((null)).
Wow... 8(
--
vda
Quoted 5 lines · Denis Vlasenko
# reiserfstune -s 1024 /dev/sdc3 # mount /dev/sdc3 /.3 -o noatime mount: you must specify the filesystem type
Quoted 8 lines
# dmesg | tail -4 br: topology change detected, propagating br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3.# reiserfsck /dev/sdc3 reiserfsck 3.6.11 (2003 www.namesys.com)
your reiserfsprogs are old. which kernel are you using?
as I can see 3.6.11 had that problem indeed, however I have no problem with progs 3.6.19 and kernel 2.6.9 even after shrinking the journal with tune 3.6.11.
Signature · 1 line
Vitaly
Quoted 12 lines · Vitaly Fertman
Quoted 11 lines · Denis Vlasenko
# reiserfstune -s 1024 /dev/sdc3 # mount /dev/sdc3 /.3 -o noatime mount: you must specify the filesystem type# dmesg | tail -4 br: topology change detected, propagating br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3.
Quoted 4 lines
Quoted 2 lines
# reiserfsck /dev/sdc3 reiserfsck 3.6.11 (2003 www.namesys.com)your reiserfsprogs are old. which kernel are you using?
# uname -a Linux pegasus 2.6.12.3-2 #1 SMP Thu Sep 15 11:04:37 EEST 2005 i686 unknown unknown GNU/Linux
Quoted 3 lines
as I can see 3.6.11 had that problem indeed, however I have no problem with progs 3.6.19 and kernel 2.6.9 even after shrinking the journal with tune 3.6.11.
Updated to reiserfsprogs-3.6.19. How to fix /dev/sdc3 now? --
vda
Quoted 13 lines · Denis Vlasenko
Quoted 12 lines · Vitaly Fertman
Quoted 11 lines · Denis Vlasenko
# reiserfstune -s 1024 /dev/sdc3 # mount /dev/sdc3 /.3 -o noatime mount: you must specify the filesystem type# dmesg | tail -4 br: topology change detected, propagating br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
# reiserfsck /dev/sdc3 reiserfsck 3.6.11 (2003 www.namesys.com)your reiserfsprogs are old. which kernel are you using?
# uname -a Linux pegasus 2.6.12.3-2 #1 SMP Thu Sep 15 11:04:37 EEST 2005 i686 unknown unknown GNU/Linux
I actually _looked at_ the uname output :) I am running wrong kernel! Rebooted into 2.6.15.1. Mount still failing.
--
vda
Quoted 146 lines · Denis Vlasenko
Quoted 26 lines · Denis Vlasenko
Quoted 7 lines · Chris Mason
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxxThe filesystem must be unmounted first.
It doesn't want to, for no obvious reason:
# mount rootfs on / type rootfs (rw) /dev/root on / type ext2 (ro,nogrpid) none on /proc type proc (rw,nodiratime) none on /sys type sysfs (rw) none on /dev type ramfs (rw) /dev/sda2 on /.share type ext2 (rw,noatime,nogrpid) /dev/sdb2 on /.1 type reiserfs (rw,noatime) /dev/sdc2 on /.2 type reiserfs (rw,noatime) /dev/sdc3 on /.3 type reiserfs (rw,noatime) /dev/sda2 on /.local type ext2 (rw,noatime,nogrpid) none on /dev/pts type devpts (rw) automount(pid1043) on /.local/mnt/auto type autofs (rw) # umount /.3Quoted 10 lines
umount: /.3: device is busy # mount -o ro,remount /.3 # umount /.3 umount: /.3: device is busy # reiserfstune /dev/sdc3 reiserfstune: Reiserfstune is not allowed to be run on mounted filesystem. # lsof -nP | grep -F '/.3' # lsof -nP | grep -F '/dev/sdc'Removed it from
/etc/fstab, after reboot:# reiserfstune -s 1024 /dev/sdc3 reiserfstune: Journal device has not been specified. Assuming journal is on the main device (/dev/sdc3).Current parameters:
Filesystem state: consistent
Reiserfs super block in block 16 on
0x823of format 3.6 with standard journal Count of blocks on the device: 2094474
Number of bitmaps: 64
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 1019710 Root block: 32941
Filesystem is cleanly umounted
Tree height: 4
Hash function used to sort names: "r5"
Objectid map size 54, max 972
Journal parameters:
Device [0x0]
Magic [0x2cac04df]
Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks
Max batch size 900 blocks
Max commit age 30
Blocks reserved by journal: 0
Fs state field:0x0:sb_version: 2
inode generation number: 293892
UUID: 7b3aa1ab-40bd-44da-98dc-b37b21f7add0
LABEL:
Set flags in SB:
ATTRIBUTES CLEAN
reiserfstune: Current journal parameters:
Device [0x0]
Magic [0x2cac04df]
Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks
Max batch size 900 blocks
Max commit age 30
WARNING: wrong transaction max size (1024). Changed to 511
reiserfstune: New journal parameters:
Device [0x0]
Magic [0x70dbc903]
Size 1024 blocks (including 1 for journal header) (first block 18) Max transaction length 511 blocks
Max batch size 449 blocks
Max commit age 30
Reiserfs super block in block 16 on0x823of format 3.6 with non-standard journal Count of blocks on the device: 2094474
Number of bitmaps: 64
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 1019710 Root block: 32941
Filesystem is cleanly umounted
Tree height: 4
Hash function used to sort names: "r5"
Objectid map size 54, max 972
Journal parameters:
Device [0x0]
Magic [0x70dbc903]
Size 1024 blocks (including 1 for journal header) (first block 18) Max transaction length 511 blocks
Max batch size 449 blocks
Max commit age 30
Blocks reserved by journal: 8193
Fs state field:0x0:sb_version: 2
inode generation number: 293892
UUID: 7b3aa1ab-40bd-44da-98dc-b37b21f7add0
LABEL:
Set flags in SB:
ATTRIBUTES CLEAN
reiserfstune: ATTENTION: YOU ARE ABOUT TO SETUP THE NEW JOURNAL FOR THE "/dev/sdc3"! AREA OF "/dev/sdc3" DEDICATED FOR JOURNAL WILL BE ZEROED!
Continue (y/n):y
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok# mount /dev/sdc3 /.3 -o noatime mount: you must specify the filesystem type# dmesg | tail -4 br: topology change detected, propagating br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3.# reiserfsck /dev/sdc3 reiserfsck 3.6.11 (2003 www.namesys.com)*************************************************************
** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. **
*************************************************************Will read-only check consistency of the filesystem on
/dev/sdc3Will put log info to 'stdout'
Quoted 8 lines
Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Failed to open the journal device ((null)).
Wow... 8(--
vda
would you try
reiserfsck -j /dev/sdc3 /dev/sdc3
Thanks,
Edward.
Quoted 107 lines · Edward Shishkin
Quoted 107 lines
# reiserfstune -s 1024 /dev/sdc3 reiserfstune: Journal device has not been specified. Assuming journal is on the main device (/dev/sdc3).Current parameters:
Filesystem state: consistent
Reiserfs super block in block 16 on
0x823of format 3.6 with standard journal Count of blocks on the device: 2094474
Number of bitmaps: 64
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 1019710 Root block: 32941
Filesystem is cleanly umounted
Tree height: 4
Hash function used to sort names: "r5"
Objectid map size 54, max 972
Journal parameters:
Device [0x0]
Magic [0x2cac04df]
Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks
Max batch size 900 blocks
Max commit age 30
Blocks reserved by journal: 0
Fs state field:0x0:sb_version: 2
inode generation number: 293892
UUID: 7b3aa1ab-40bd-44da-98dc-b37b21f7add0
LABEL:
Set flags in SB:
ATTRIBUTES CLEAN
reiserfstune: Current journal parameters:
Device [0x0]
Magic [0x2cac04df]
Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks
Max batch size 900 blocks
Max commit age 30
WARNING: wrong transaction max size (1024). Changed to 511
reiserfstune: New journal parameters:
Device [0x0]
Magic [0x70dbc903]
Size 1024 blocks (including 1 for journal header) (first block 18) Max transaction length 511 blocks
Max batch size 449 blocks
Max commit age 30
Reiserfs super block in block 16 on0x823of format 3.6 with non-standard journal Count of blocks on the device: 2094474
Number of bitmaps: 64
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 1019710 Root block: 32941
Filesystem is cleanly umounted
Tree height: 4
Hash function used to sort names: "r5"
Objectid map size 54, max 972
Journal parameters:
Device [0x0]
Magic [0x70dbc903]
Size 1024 blocks (including 1 for journal header) (first block 18) Max transaction length 511 blocks
Max batch size 449 blocks
Max commit age 30
Blocks reserved by journal: 8193
Fs state field:0x0:sb_version: 2
inode generation number: 293892
UUID: 7b3aa1ab-40bd-44da-98dc-b37b21f7add0
LABEL:
Set flags in SB:
ATTRIBUTES CLEAN
reiserfstune: ATTENTION: YOU ARE ABOUT TO SETUP THE NEW JOURNAL FOR THE "/dev/sdc3"! AREA OF "/dev/sdc3" DEDICATED FOR JOURNAL WILL BE ZEROED!
Continue (y/n):y
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok# mount /dev/sdc3 /.3 -o noatime mount: you must specify the filesystem type# dmesg | tail -4 br: topology change detected, propagating br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3.# reiserfsck /dev/sdc3 reiserfsck 3.6.11 (2003 www.namesys.com)*************************************************************
** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. **
*************************************************************Will read-only check consistency of the filesystem on
/dev/sdc3Will put log info to 'stdout'Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Failed to open the journal device ((null)).
Quoted 8 lines
Quoted 5 lines
Wow... 8(--
vdawould you try
reiserfsck -j/dev/sdc3/dev/sdc3
Updated tools to reiserfsprogs-3.6.19, then:
# reiserfsck -j /dev/sdc3 /dev/sdc3 reiserfsck 3.6.19 (2003 www.namesys.com)
*************************************************************
** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. **
*************************************************************
Will read-only check consistency of the filesystem on /dev/sdc3 Will put log info to 'stdout'
Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Wed Feb 1 16:24:20 2006
###########
Replaying journal..
No transactions found
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
Leaves 8075
Internal nodes 52
Directories 1792
Other files 31865
Data block pointers 1058363 (0 of them are zero)
Safe links 0
###########
reiserfsck finished at Wed Feb 1 16:26:33 2006
###########
--
vda
Quoted 32 lines · Denis Vlasenko
Updated tools to reiserfsprogs-3.6.19, then:
# reiserfsck -j /dev/sdc3 /dev/sdc3 reiserfsck 3.6.19 (2003 www.namesys.com)*************************************************************
** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. **
*************************************************************Will read-only check consistency of the filesystem on
/dev/sdc3Will put log info to 'stdout'Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Wed Feb 1 16:24:20 2006
###########
Replaying journal..
No transactions found
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
Leaves 8075
Quoted 8 lines
Internal nodes 52
Directories 1792
Other files 31865
Data block pointers 1058363 (0 of them are zero) Safe links 0
###########
reiserfsck finished at Wed Feb 1 16:26:33 2006
###########
I hit [reply] too fast.
reiserfsck 3.6.19 reports success now even without -j option, but still (kernel 2.6.15.1):
# mount /dev/sdc3 /.3 mount: you must specify the filesystem type
How can I fix it?
--
vda
Quoted 2 lines · Denis Vlasenko
# mount /dev/sdc3 /.3 mount: you must specify the filesystem type
what happens if you actually specify the fs type?
Gruss
Bernd
Quoted 5 lines · Bernd Eckenfels
Quoted 2 lines · Denis Vlasenko
# mount /dev/sdc3 /.3 mount: you must specify the filesystem typewhat happens if you actually specify the fs type?
It works.
# mount -t reiserfs /dev/sdc3 /.3 # umount /.3 # mount /dev/sdc3 /.3 mount: you must specify the filesystem type
--
vda
Quoted 14 lines · Chris Mason
Quoted 8 lines
Quoted 8 lines
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
/dev/sdc3on /.3 type reiserfs (rw,noatime)causes oom kill storm. "ls -lR", "find ." etc work fine.
In order for the journaled filesystems to make sure the FS is consistent after a crash, we need to keep some blocks in memory until other blocks have been written. These blocks are pinned, and can't be freed until a certain amount of io is done.
Quoted 8 lines
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxx
I had reiserfsprogs 3.6.11 and reiserfstune (above command) made my /dev/sdc3 unmountable without -t reiserfs. I upgraded reiserfsprogs to 3.6.19 and now reiserfsck /dev/sdc3 reports no problems, but mount problem persists:
# mount -t reiserfs /dev/sdc3 /.3 # umount /.3 # mount /dev/sdc3 /.3 mount: you must specify the filesystem type # dmesg | tail -3 br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3.
"chown -Rc <n>:<m> ." now does not OOM kill the box, so this issue is resolved, thanks!
Can I restore sdc3 somehow that I won't need -t reiserfs in mount command? You can find result of
dd if=/dev/sdc3 of=1m bs=1M count=1
at http://195.66.192.167/linux/1m
--
vda
Quoted 18 lines
Quoted 1 line
reiserfstune -s 1024
/dev/xxxxI had reiserfsprogs 3.6.11 and reiserfstune (above command) made my
/dev/sdc3unmountable without -t reiserfs. I upgraded reiserfsprogs to 3.6.19 and now reiserfsck/dev/sdc3reports no problems, but mount problem persists:# mount -t reiserfs /dev/sdc3 /.3 # umount /.3 # mount /dev/sdc3 /.3 mount: you must specify the filesystem type # dmesg | tail -3 br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3."chown -Rc <n>:<m> ." now does not OOM kill the box, so this issue is resolved, thanks!
Quoted 6 lines
Can I restore sdc3 somehow that I won't need -t reiserfs in mount command? You can find result of
dd if=/dev/sdc3 of=1m bs=1M count=1
the problem seems to be in mount program, which version do you use? I still have no problem with your 1m image, mount version is 2.11z, 2.12c.
Signature · 1 line
Vitaly
Quoted 21 lines · Vitaly Fertman
Quoted 21 lines
Quoted 1 line
reiserfstune -s 1024
/dev/xxxxI had reiserfsprogs 3.6.11 and reiserfstune (above command) made my
/dev/sdc3unmountable without -t reiserfs. I upgraded reiserfsprogs to 3.6.19 and now reiserfsck/dev/sdc3reports no problems, but mount problem persists:# mount -t reiserfs /dev/sdc3 /.3 # umount /.3 # mount /dev/sdc3 /.3 mount: you must specify the filesystem type # dmesg | tail -3 br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3."chown -Rc <n>:<m> ." now does not OOM kill the box, so this issue is resolved, thanks!
Can I restore sdc3 somehow that I won't need -t reiserfs in mount command? You can find result of
Quoted 7 lines
Quoted 3 lines
dd if=/dev/sdc3 of=1m bs=1M count=1
the problem seems to be in mount program, which version do you use? I still have no problem with your 1m image, mount version is 2.11z, 2.12c.
# mount --version mount: mount-2.11p
Obviously you are right, mount reads some data in /dev/sdc3 itself and decides to try specific fs types first instead of just asking kernel to autodetect:
...
...stat64("/dev/sdc3", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 35), ...}) = 0 open("/dev/sdc3", O_RDONLY|O_LARGEFILE) = 4_llseek(4, 0, [0], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 576) = 576_llseek(4, 512, [512], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512_llseek(4, 1024, [1024], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 228) = 228_llseek(4, 1024, [1024], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24_llseek(4, 3072, [3072], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512_llseek(4, 8192, [8192], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1376) = 1376_llseek(4, 8192, [8192], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 64) = 64
_llseek(4, 8192, [8192], SEEK_SET) = 0 read(4, "\0\0\0\0\0\0\0\0", 8) = 8 _llseek(4, 32768, [32768], SEEK_SET) = 0
read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 112) = 112_llseek(4, 32768, [32768], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048) = 2048_llseek(4, 65536, [65536], SEEK_SET) = 0read(4, "\212\365\37\0>\217\17\0\255\200\0\0\22\0\0\0\0\0\0\0\377"..., 64) = 64_llseek(4, 0, [0], SEEK_SET) = 0read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
close(4) = 0open("/etc/filesystems", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/proc/filesystems", O_RDONLY|O_LARGEFILE) = 4fstat64(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f7f000 read(4, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 277 mount("/dev/sdc3", "/.3", "msdos", 0xc0ed0000, 0) = -1 EINVAL (Invalid argument) mount("/dev/sdc3", "/.3", "vfat", 0xc0ed0000, 0) = -1 EINVAL (Invalid argument)
read(4, "", 1024) = 0 close(4) = 0 munmap(0xb7f7f000, 4096) = 0
rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV], NULL, 8) = 0 open("/usr/share/locale/ru_RU.KOI8-R/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/ru_RU.koi8r/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/ru_RU/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/ru.KOI8-R/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/ru.koi8r/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/ru/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory) write(2, "mount: you must specify the file"..., 44) = 44
mount from busybox 1.0 works fine:
# strace busybox mount /dev/sdc3 /.3
execve("/sbin/busybox", ["busybox", "mount", "/dev/sdc3", "/.3"], [/* 24 vars */]) = 0
ioctl(0, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
getuid() = 0
getgid() = 0
getuid() = 0
getgid() = 0
setgid(0) = 0
setuid(0) = 0
brk(0) = 0x80f6000
brk(0x80f7000) = 0x80f7000
brk(0x80f8000) = 0x80f8000
brk(0x80f9000) = 0x80f9000
stat64("/dev/sdc3", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 35), ...}) = 0
open("/etc/filesystems", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/proc/filesystems", O_RDONLY|O_LARGEFILE) = 4
ioctl(4, SNDCTL_TMR_TIMEBASE, 0xbfb32740) = -1 ENOTTY (Inappropriate ioctl for device)
brk(0x80fa000) = 0x80fa000
read(4, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 4096) = 277
mount("/dev/sdc3", "/.3", "reiserfs", 0xc0ed0000, 0x80f6008) = 0
close(4) = 0
_exit(0)--
vda
Quoted 36 lines · Denis Vlasenko
Quoted 9 lines · Chris Mason
Quoted 8 lines
Quoted 8 lines
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
/dev/sdc3on /.3 type reiserfs (rw,noatime)causes oom kill storm. "ls -lR", "find ." etc work fine.
Quoted 7 lines
In order for the journaled filesystems to make sure the FS is consistent after a crash, we need to keep some blocks in memory until other blocks have been written. These blocks are pinned, and can't be freed until a certain amount of io is done.
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxxI had reiserfsprogs 3.6.11 and reiserfstune (above command) made my
/dev/sdc3unmountable without -t reiserfs. I upgraded reiserfsprogs to 3.6.19 and now reiserfsck/dev/sdc3reports no problems, but mount problem persists:# mount -t reiserfs /dev/sdc3 /.3 # umount /.3 # mount /dev/sdc3 /.3 mount: you must specify the filesystem type # dmesg | tail -3 br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3."chown -Rc <n>:<m> ." now does not OOM kill the box, so this issue is resolved, thanks!
Quoted 6 lines
Can I restore sdc3 somehow that I won't need -t reiserfs in mount command? You can find result of
dd if=/dev/sdc3 of=1m bs=1M count=1
At the risk of stating the obvious:
1 - is reaser a module, and is it loaded?
2 - did this ever work? I think you said you removed the entry from fstab, was the filetype there which made it work?
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 21 lines · Bill Davidsen
Quoted 21 lines
Quoted 1 line
reiserfstune -s 1024
/dev/xxxxI had reiserfsprogs 3.6.11 and reiserfstune (above command) made my
/dev/sdc3unmountable without -t reiserfs. I upgraded reiserfsprogs to 3.6.19 and now reiserfsck/dev/sdc3reports no problems, but mount problem persists:# mount -t reiserfs /dev/sdc3 /.3 # umount /.3 # mount /dev/sdc3 /.3 mount: you must specify the filesystem type # dmesg | tail -3 br: port 1(ifi) entering forwarding state FAT: bogus number of reserved sectors VFS: Can't find a valid FAT filesystem on dev sdc3."chown -Rc <n>:<m> ." now does not OOM kill the box, so this issue is resolved, thanks!
Can I restore sdc3 somehow that I won't need -t reiserfs in mount command? You can find result of
Quoted 8 lines
Quoted 3 lines
dd if=/dev/sdc3 of=1m bs=1M count=1
At the risk of stating the obvious:
1 - is reaser a module, and is it loaded?
2 - did this ever work? I think you said you removed the entry from fstab, was the filetype there which made it work?
1 - not a module, 2 - fstab line was "/dev/sdc3 /.3 auto noatime,rw 1 1". But anyway. mount problem is solved now, mount from util-linux 2.11p tried to "autodetect" fs and thought it's a FAT partition.
mount from busybox 1.0 works.
--
vda
Hi!
Quoted 27 lines
Quoted 9 lines
Quoted 9 lines
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
Quoted 8 lines
Quoted 8 lines
/dev/sdc3on /.3 type reiserfs (rw,noatime)causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills while chown's RSS stays below 4MB.
In order for the journaled filesystems to make sure the FS is consistent after a crash, we need to keep some blocks in memory until other blocks have been written. These blocks are pinned, and can't be freed until a certain amount of io is done.
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
Quoted 6 lines
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxxThe filesystem must be unmounted first.
Could we refuse to mount filesystem unless journal_size < physmem_size/2 or something like that?
I was not aware of this trap, and it seems unlikely that users know about it...
Pavel
Signature · 1 line
Thanks, Sharp!
Quoted 9 lines · Pavel Machek
Quoted 9 lines
In the case of reiserfs, it might pin as much as the size of the journal at any time. The default journal is 32MB, which is much too large for a system with only 32MB of ram.
You can shrink the log of an existing filesystem. The minimum size is 513 blocks, you might try 1024 as a good starting poing.
reiserfstune -s 1024
/dev/xxxx
Quoted 7 lines
Quoted 1 line
The filesystem must be unmounted first.
Could we refuse to mount filesystem unless
journal_size< physmem_size/2 or something like that?I was not aware of this trap, and it seems unlikely that users know about it...
Maybe reiserfs code should use journal of reduced size on lowmem boxes. Basically "reiserfstune -s 1024 /dev/xxxx" on-the-fly.
--
vda
Quoted 2 lines · Denis Vlasenko
Maybe reiserfs code should use journal of reduced size on lowmem boxes. Basically "reiserfstune -s 1024
/dev/xxxx" on-the-fly.
I posted something about that, but you really need to cover the case where there are multiple mounts and the sum of all journal pins is kept reasonable.
See my earlier post if you care.
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 11 lines · Hans Reiser
Quoted 9 lines · Denis Vlasenko
[CCing namesys]
Narrowed it down to 100% reproducible case:
chown -Rc 0:<n> .in a top directory of tree containing ~21938 files on reiser3 partition:
Quoted 7 lines
Quoted 7 lines
/dev/sdc3on /.3 type reiserfs (rw,noatime)causes oom kill storm. "ls -lR", "find ." etc work fine.
I suspected that it is a leak in winbindd libnss module, but chown does not seem to grow larger in top, and also running it under softlimit -m 400000 still causes oom kills
(typo, must be -m 40000000)
Quoted 3 lines
Quoted 1 line
while chown's RSS stays below 4MB.
Chris, would you like to handle this?
fs seems to be ok fsck-wise:
# mount -o remount,ro /.3 # reiserfsck /dev/sdc3 reiserfsck 3.6.11 (2003 www.namesys.com)
*************************************************************
** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. **
*************************************************************
Will read-only check consistency of the filesystem on /dev/sdc3 Will put log info to 'stdout'
Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Mon Jan 30 14:11:15 2006
###########
Filesystem seems mounted read-only. Skipping journal replay. Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
Leaves 8075
Internal nodes 52
Directories 1792
Other files 31865
Data block pointers 1058363 (0 of them are zero)
Safe links 0
###########
reiserfsck finished at Mon Jan 30 14:13:28 2006
###########
However, there is one strange thing: I cannot umount it. Why? There is no open files.
# umount /.3 umount: /.3: device is busy # lsof -nP | grep -F '/.3' # lsof -nP | grep -F 'sdc' # uname -a Linux pegasus 2.6.14.6 #1 SMP Mon Jan 30 08:46:20 EET 2006 i686 unknown unknown GNU/Linux
--
vda
Quoted 4 lines · Denis Vlasenko
# umount /.3 umount: /.3: device is busy # lsof -nP | grep -F '/.3' # lsof -nP | grep -F 'sdc'
You can try "lsof +f -- /.3" also
Gruss
Bernd
Quoted 7 lines · Bernd Eckenfels
Quoted 4 lines · Denis Vlasenko
# umount /.3 umount: /.3: device is busy # lsof -nP | grep -F '/.3' # lsof -nP | grep -F 'sdc'You can try "lsof +f -- /.3" also
Didn't help, but later I found it.
Kernel nfsd daemon was keeping it busy.
It's not visible in lsof.
--
vda