linux disk access when idle

The scope of this question is probably broader than Linux RAID, but I thought I'd try here before going to the LKML...

I have a fileserver/NAS box that has four Western Digital GreenPower drives in software RAID-5 (i.e. Linux md). There was some noise about these drives not too long ago: when used under Linux, they tend to have really high (and rapidly increasing) SMART Load_Cycle_Count values[1].

Basically, in order to save power, these drives park the read/write heads after so many seconds of inactivity. According to SilentPCReview[2], the 1 TB model uses 5.7W when idle and only 3.7W when the heads are unloaded. Apparantly, each head parking event causes teh Load_Cycle_Count value to increase.

Most people seek to disable this head parking behavior to stop that Load_Cycle_Count value from growing alarmingly high. I actually want to use it to it's full potential.

I've got a Kill-A-Watt electricity meter connected to my NAS box. I see a difference of 10 watts power usage when the heads park (I can hear them unload, so I know when it happens). Allowing for some power supply inefficiency, I'd say my observation is consistent with SPCR's numbers.

The problem is, the drives don't stay in this parked state very long. I haven't actually timed the state changes, but average power consumption over a long time (e.g. a week or more) is at the higher (i.e. +10) level.

So I figure, something is accessing these drives shortly after the heads park, causing them to un-park (and increasing power consumption). But this machine is idle 95% of the time. And even then, the overwhelming majority of the accesses are reads, with very few writes (literally, a handful of writes per week).

So what I'm trying to figure out is, what is causing the disk access? It could be any one of:

- Kernel
- RAID subsystem (i.e. md)
- XFS filesystem
- NFS
- Samba
- ???

I'm hoping someone has enough knowledge of these systems to point me in the right direction for tuning things. The goal is that when the machine is idle, it is "truly" idle, meaning, no disk accesses take place and the heads can stay parked (thus saving energy).

Thank you,
Matt

LINKS:
[1] http://groups.google.com/group/linux.kernel/browse_thread/thread/505ccf760023d132/7e4f4e996f911efd

[2] http://www.silentpcreview.com/article804-page2.html

Are you running smartd? It polls the drives every 30minutes.

Regards,

Richard

I had a similar issue, as I wanted to take full advantage of parking heads but they kept coming alive again. I was putting them to sleep with hdparm.

I gave up at the end, but thought I had read that it was likely syslog or other internals that were polling the disc too frequently. Sorry Im a bit vague on the details.

If this was the case (I didn't really investigate) then I thought, stuff it, because I wanted syslog to keep doing it's thing.

--andrew

I did this with the data drives in my array - never managed it with the system drives though.

You need to tell smartd to not poll if the drives are asleep and also tell syslog not to poll too often.

There is some for of flag you can apply to see write requests and the application that is doing them.......?

BUT a curious problem I have - and have NEVER really solved is that when a drive is asleep if a scheduled smart check runs - the drive doesn't wake up quickly enough so smart brands the drive as 'incapable of a smart test' and sends me a warning email.... every single time...

-----------------------
N: Jon Hardcastle
E: [email protected]
'Do not worry about tomorrow, for tomorrow will bring worries of its own.'
-----------------------

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

You might find this useful:

echo 1 > /proc/sys/vm/block_dump

Then watch your syslog files...

Brian

I was running smartd, until I received your post :)

Unfortunately, this does not help keep the heads parked any longer.

Nonetheless, thank you for the hint.

Matt

I think that you can use blktrace to see who is issuing the IO's that spin up your drives....

ric

Good to know. I should have mentioned, however, that my RAID5 drives are just for data storage. The OS (and therefore syslog and friends) is actually on a compact flash card (using one of those compact flash to IDE converters).

Thanks,
Matt

Thank you. That along with Ric Wheeler's suggestion of blktrace ought to help me get to the bottom of this.

Thanks again!
Matt

/proc/sys/vm/dirty_expire_centisecs still set to 30 sec?

bill davidsen <[email protected]>
 CTO TMR Associates, Inc

"You are disgraced professional losers. And by the way, give us our money back."
   - Representative Earl Pomeroy,  Democrat of North Dakota
on the A.I.G. executives who were paid bonuses  after a federal bailout.

$ cat /proc/sys/vm/dirty_expire_centisecs
2999

Yup. Maybe I don't understand exactly what this does, but if the drives are idle (i.e. no reads or writes) for say an hour or more, why would there be any dirty data to flush?

That's why I'm confused---this machine literally goes for hours at a time without and read or write attemps on these drives.

-Matt

I haven't followed this whole thread, but there is some aspect of ext3 journalling which hits that disk with regularity. Obviously not a write flush, but some sort of check out of insecurity or something.

I thought it was just a trait of the animal. My Linux systems have always done disk access every few seconds, 24/7. From '92 on. b-

Not necessarily. I don't know what they did to achieve it, but Netgear's ReadyNAS devices can be configured to park the drives when they're left alone, it works (unless you're using SqueezeCenter), and they're Linux-based.

Cheers,

John.

As I understand it, the ReadyNAS uses XFS.

Regards,

Richard

For one, syslogd.
It's probably worthwhile to look up information for laptop power saving. In particular, laptop-mode in the kernel, and laptop-mode-tools:
http://samwel.tk/laptop_mode
I use this on my laptop, and I don't think I ever have a "surprise" disk spinup. The whole laptop-mode-tools setup is probably much more than you want, but it's a good place to start. The FAQ there also mentions how to get syslog to stop syncing to disk.
You may also find 'lm-profiler' useful - this will tell you the which process is causing disk accesses. Finally, for general power saving, 'powertop' tracks the power state information for your CPU, and will list the processes that are causing your CPU to come out of any sleep/idle states.

Just a data point: I've got a box (Debian Lenny) with five disks, system disk plus 4-disk RAID5 array, used as a backup server (rsnapshot), and I power the disks in the raid array down after running the backup - and they stay powered down, even though I don't umount the filesystems let alone stop the array. When accessed (like for restoring something from the backup), they wake up, so I've got an hourly cron job powering them down again - but most days (when there're no restores) they are powered up only during the backup run.

So it can be done. It's not filesystem type dependent either, there's one jfs and one ext3 in the array.

The system disk stays up, though. Some experimenting suggested that getting it to stay down would require putting /tmp and parts of /var on a ramdisk (/var/log at least), but I didn't go to the trouble of tracking down all disk-awakening services.

Tapani Tarvainen

Smartd has an option not to wake up disks. If you have smartd scann less often than the suspend time of the disk then idle disk can spinn down and stay that way.

MfG
Goswin

Same here. I have a raid1 for the system on 2 drives and a raid5 over 6 drives for data. The raid5 is ext3 powers down after 15 minutes without access. The raid1 stays up all the time due to /var/log/syslog being written to too often. /tmp is tmpfs though.

MfG
Goswin

For five hours today, I ran blktrace on md1 as well as the drives that make up md1 (sd[a-d]). I also enabled block dump via

echo 1 > /proc/sys/vm/block_dump

Nothing. Neither block_dump nor blktrace recorded anything on md1 or sd[abcd]. I caught one instance where the heads parked (I both heard them park as well as saw the power meter show the 10 watt drop). They only stayed parked for about five minutes. Yet neither blktrace nor block_dump caught anything.

I'm going to try lm-profiler now (from the laptop-mode-tools package), but I have low expectations.

I'm beginning to think that it's not even direct access to the drives that cause the heads to unpark, e.g. perhaps even activity on my system drive (a compact flash card connected to the PATA port) prevents the data drives from parking their heads.

Are you doing regular self test? I'm not sure if non-io connections would show using those tools, I just haven't tried them at that level.

bill davidsen <[email protected]>
 CTO TMR Associates, Inc

"Now we have another quarterback besides Kurt Warner telling us during postgame
interviews that he owes every great thing that happens to him on a football
field to his faith in Jesus. I knew there had to be a reason why the Almighty
included a mute button on my remote."
			-- Arthur Troyer on Tim Tebow (Sports Illustrated)

What's in your cronjob?

This is correct. One thing I noticed yesterday, when I was running blktrace and having block_dump enabled: the heads *never* parked during this time. Even though I had zero accesses on the data drives, the system drive was getting hit pretty heavily (because I deliberately had syslog write the block_dump stuff). When I finally stopped all the heavy logging, I ran "sync" and noticed that a few seconds later, the heads parked.

Almost exactly five minutes later, the heads un-parked.

I repeated: I ran the "sync" command, the heads unloaded, five minutes later, they re-loaded. I repeated this exactly several times.

So I re-enabled block_dump (i.e. "echo 1 > /proc/sys/vm/block_dump"), but disabled syslog (so the system drive wouldn't be thrashed). I ran the "sync, then wait five minutes test": pdflush and kjournald apparently caused the wakeup after five minutes.

I set my system drive to ext2 (instead of ext3) in /etc/fstab and rebooted. I was under the impression that using ext2 instead of ext3 would eliminate the need for kjournald, but this doesn't appear to be the case.

So, based on what I've seen so far, there's some "five minute thing" going on that involves pdflush and/or kjournald.

-Matt