IOzone testing

Good morning,

I'm running IOzone on a NILFS2 partition and it's running alot slower that I had expected. I'm pretty sure it's because of the checkpoints. I can change the protection_period to something very small (or zero). Would that improve the performance?

In addition, is there a way to turn off checkpointing?

Thanks!

Jeff

The protection period only affects how the cleaner daemon will clean the old checkpoints, if 0 I guess it will just clean them up as soon as it can but it won't stop creating them. You could, just for benchmark purpose, kill nilfs_cleanerd which will stop checkpoint related activities.

The whole filesystem is based on those checkpoint, the fact that it is a log filesystem makes it so all writes are made at the end of the device, and when a write is complete, a checkpoint is made.

It will only stop the clean up of checkpoints outside of policy - correct? So it won't stop them from being created but it will stop the cleaning process.

Understood. But for benchmarks like IOzone where you are creating files and most likely syncing, this will trigger a checkpoint. I haven't looked too much at IOzone but it might even be causing a sync after each record right, again forcing a checkpoint.

So I was looking for some way to control how often checkpoints are created - something like the protection_period option for the GC. But I understand that the checkpoint is a part of the design of the file system.

Thanks!

Jeff

It will completely stop cleaning checkpoints, which is what is consuming drive I/O. Checkpoints in themselve don't use much ressources, cleaning them however does.

Checkpoints are created each time a change is commited to disk right, but creating a checkpoint does not take much system ressources, try time'ing mkcp, it won't use any ressources, it just writes a marker at the end of the disk to say that all that is written behind it is good to use if a crash happens or the user wants to mount the partition in the past. Increasing checkpoint period would just increases the risk of losing more data on crash with minimal or no performance gain.

I don't know if you ever worked with shadow copies in Windows but creating a shadow copie would consume lots of drive I/O and CPU, that is because NTFS wasn't conceived as a log filesystem like NILFS and must write structures and files to the disk for whatever changes happened to the disk, it must also freeze the filesystem and make sure all writes are atomic before creating the snapshot. While NILFS just write a checkpoint each time it knows the last write is complete, converting a checkpoint to a snapshot then is just a matter of changing a flag so the GC won't remove it afterward.

Got it - thanks.

At the same time, I also hit an out of space message. Why would I get this message? If the checkpoints are just creating a small maker then it shouldn't take much space. Unless of course, a huge number of checkpoints are created.

Thanks!

If nilfs_cleanerd is shut down, then the filesystem will fill up until there's no room for more data. When you delete data, only the nilfs_cleanerd can reclaim the space, which is why you set up a protection_period, deleted data will never be overwritten unless reclaimed by the cleanerd.
Here is a program I made to represent the disk in a graphical form, you'll see the cleanerd in action:
http://ticpu.net:8080/~jerome/lssu-gtk.py

You need python-2.4+ with pygtk and nilfs-utils to run the program.

Cool program - I will definitely try this.

So, let me summarize what I think I should do to test nilfs with IOzone without the impact of the checkpoints and without filling up the space.

- Run IOzone without cleanerd running to minimize the impact of checkpoints on IO performance.
- Assuming I don't fill up the file system, after a run of IOzone, I need to run cleanerd but put the protection period to something very small (e.g. 1 second) to reclaim the space.
- Shut down cleanerd again and run another IOzone test. (rinse and repeat)

Does this sound about right?

Thanks!

Jeff

That sounds right, either run cleanerd or mkfs the partition again for faster results ;)

Good point about using mkfs. :)

I think I'm going to run two test: (1) using the process above, (2) setting the protection period to 1 (maybe 0). Let's see what the differences are.

Thanks!

Jeff

Be aware that garbage collection will take quite a while on a full drive to go back to 0 unless you also increase the segments cleaned per run and if you accidentally leave it high it will impact performance.

This is why I had asked about a way to force a full GC with a protection period other than that in the conf file. It seems like quick forced cleanup when a user deletes a lot of files is a common need...

Byron

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

That is a good feature to add - quick full GC with a different protection period. +1 on that option.

I think I might just to a mkfs in between runs to make sure that the checkpoints are eliminated. That works for my testing but it doesn't work for real, live, working file systems.

Jeff

Hi
I'm running SLES 11, 2.6.27.19-5-default with NILFS2 nilfs-2.0.16. I have a 1.5Tb NILFS2 partition which I am setting up with the intention of using Robocopy from various PCs via samba. The robocopy scripts run nightly and a checkpoint is taken once night. A script stops samba, unmounts the previous weeks checkpoint, deletes the checkpoint, creates a new one and then mounts it and restarts samba. This should mean that at any time the user can go back to 'snapshot_{DAY}' to get their files back.

So far so good.

However as I copy the previously backed up files from the previous linux machine where I was doing this (only giving a 'current' copy with reiserfs). I'm finding that the new machine is occasionally hanging. The OS just locks up, screen on console frozen but host still responds to ping.

I'm trying to work out what is causing the hang, I'm getting various messages in the log from smartd relating to the disk which houses the NILFS along the lines of:

 Oct  2 09:56:59 cpli6008 syslog-ng[1933]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=947', processed='center(received)=478', processed='destination(newsnotice)=0', processed='destination(acpid)=0', processed='destination(firewall)=0', processed='destination(mail)=12', processed='destination(mailinfo)=12', processed='destination(console)=151', processed='destination(newserr)=0', processed='destination(newscrit)=0', processed='destination(messages)=466', processed='destination(mailwarn)=0', processed='destination(localmessages)=0', processed='destination(netmgm)=0', processed='destination(mailerr)=0', processed='destination(xconsole)=151', processed='destination(warn)=155', processed='source(src)=478'
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 110 to 112
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 117
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 189 High_Fly_Writes changed from 88 to 87
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 40 to 39
Oct  2 09:57:25 cpli6008 smartd[3473]: Device: /dev/sdb [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 50 to 51

{machine stops responding and gets power cycled}

Oct  2 10:10:58 cpli6008 syslog-ng[1948]: syslog-ng starting up; version='2.0.9'

Do folks think that the hang is NILFS or dodgy hardware/reporting from smartd? Is there any advise on getting some debug or status information from NILFS to help show it isn't the cause of the problem. I would have expected that if it went bang I'd have seen something 'worrying' in the log.

For information the hardware is a Dell Precision 380.

Many thanks
David Barham
Siemens PLM Software

Hi,

The nilfs2 standalone module has a debug mode. You can enable it by commenting out the following line (i.e. CONFIG_NILFS_DEBUG=y) in nilfs2-module/fs/Makefile before compiling:

ifndef CONFIG_NILFS
EXTERNAL_BUILD=y
CONFIG_NILFS=m
# Uncomment below to do debug build. CONFIG_NILFS_DEBUG=y
# Uncomment below to enable bmap validity check. #CONFIG_NILFS_BMAP_DEBUG=y
endif

By the way, I'm planning to release nilfs-2.0.17 tomorrow in order to solve file system corruption problems which infrequently happen and were reported on this list.

The bugfix was already merged in the mainline and also sent to -stable trees for 3.6.30 and 3.6.31, but not yet done.

Your problem looks hardware problem to me, but I think the new version is worth a try.

Cheers,
Ryusuke Konishi

I concur. Please, open the computer and touch the hard disk to check whether that reading is correct or not. If it is, your disk is dead or about to die.

I checked the hardware today as the machine keeps falling over, but only as I'm doing data copies into the nilfs2 filesystem via NFS. The last few times I've not had any errors in the logs at all except those the clearerd which is in debug mode and the startup after I power cycle the machine.

I tried to build a debug version of 2.0.17 but I'm getting

/usr/local/sources/nilfs-2.0.17 # make
make -C fs
make[1]: Entering directory `/usr/local/sources/nilfs-2.0.17/fs'
make -C /lib/modules/2.6.27.19-5-default/build SUBDIRS=/usr/local/sources/nilfs-2.0.17/fs BUILD_DIR=/usr/local/sources/nilfs-2.0.17/fs modules make[2]: Entering directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make -C ../../../linux-2.6.27.19-5 O=/usr/src/linux-2.6.27.19-5-obj/x86_64/default/. modules
CC [M] /usr/local/sources/nilfs-2.0.17/fs/debug.o
/usr/local/sources/nilfs-2.0.17/fs/debug.c: In function âsnprint_page_flagsâ:
/usr/local/sources/nilfs-2.0.17/fs/debug.c:408: error: implicit declaration of function âPagePinnedâ
make[5]: *** [/usr/local/sources/nilfs-2.0.17/fs/debug.o] Error 1
make[4]: *** [_module_/usr/local/sources/nilfs-2.0.17/fs] Error 2
make[3]: *** [sub-make] Error 2
make[2]: *** [all] Error 2
make[2]: Leaving directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/usr/local/sources/nilfs-2.0.17/fs'
make: *** [fs] Error 2

I get the same error trying to make 2.0.16.

I've had a look at the C but I'm out of my depth.

Any help would be great.

Thanks
David Barham
Siemens PLM Software

-----Original Message-----
From: Ryusuke Konishi [mailto:[email protected]]
Sent: 02 October 2009 12:42
To:; Barham, David
Subject: Re: [NILFS users] NILFS hanging SLES 11 - advise on diagnosis needed

Hi,

The nilfs2 standalone module has a debug mode. You can enable it by commenting out the following line (i.e. CONFIG_NILFS_DEBUG=y) in nilfs2-module/fs/Makefile before compiling:

ifndef CONFIG_NILFS
EXTERNAL_BUILD=y
CONFIG_NILFS=m
# Uncomment below to do debug build. CONFIG_NILFS_DEBUG=y
# Uncomment below to enable bmap validity check. #CONFIG_NILFS_BMAP_DEBUG=y
endif

By the way, I'm planning to release nilfs-2.0.17 tomorrow in order to solve file system corruption problems which infrequently happen and were reported on this list.

The bugfix was already merged in the mainline and also sent to -stable trees for 3.6.30 and 3.6.31, but not yet done.

Your problem looks hardware problem to me, but I think the new version is worth a try.

Cheers,
Ryusuke Konishi

I managed to get the debug option to compile but only after I commented out the lines in debug.c

/*#if (LINUX_VERSION_CODE > KERNEL_VERSION(2, 6, 22))

        TEST_PAGE_FLAG(page, Pinned, b, size, n, len);
        TEST_PAGE_FLAG(page, Readahead, b, size, n, len);

#endif
/*

Is there perhaps something different about 2.6.27?

Do I know need to update the mount options with a 'v' ?

Thanks
David

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Barham, David
Sent: 06 October 2009 15:24
To: [email protected]
Subject: Re: [NILFS users] NILFS hanging SLES 11 - advise on diagnosis needed

I checked the hardware today as the machine keeps falling over, but only as I'm doing data copies into the nilfs2 filesystem via NFS. The last few times I've not had any errors in the logs at all except those the clearerd which is in debug mode and the startup after I power cycle the machine.

I tried to build a debug version of 2.0.17 but I'm getting

/usr/local/sources/nilfs-2.0.17 # make
make -C fs
make[1]: Entering directory `/usr/local/sources/nilfs-2.0.17/fs'
make -C /lib/modules/2.6.27.19-5-default/build SUBDIRS=/usr/local/sources/nilfs-2.0.17/fs BUILD_DIR=/usr/local/sources/nilfs-2.0.17/fs modules make[2]: Entering directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make -C ../../../linux-2.6.27.19-5 O=/usr/src/linux-2.6.27.19-5-obj/x86_64/default/. modules
CC [M] /usr/local/sources/nilfs-2.0.17/fs/debug.o
/usr/local/sources/nilfs-2.0.17/fs/debug.c: In function âsnprint_page_flagsâ:
/usr/local/sources/nilfs-2.0.17/fs/debug.c:408: error: implicit declaration of function âPagePinnedâ
make[5]: *** [/usr/local/sources/nilfs-2.0.17/fs/debug.o] Error 1
make[4]: *** [_module_/usr/local/sources/nilfs-2.0.17/fs] Error 2
make[3]: *** [sub-make] Error 2
make[2]: *** [all] Error 2
make[2]: Leaving directory `/usr/src/linux-2.6.27.19-5-obj/x86_64/default'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/usr/local/sources/nilfs-2.0.17/fs'
make: *** [fs] Error 2

I get the same error trying to make 2.0.16.

I've had a look at the C but I'm out of my depth.

Any help would be great.

Thanks
David Barham
Siemens PLM Software

-----Original Message-----
From: Ryusuke Konishi [mailto:[email protected]]
Sent: 02 October 2009 12:42
To:; Barham, David
Subject: Re: [NILFS users] NILFS hanging SLES 11 - advise on diagnosis needed

Hi,

The nilfs2 standalone module has a debug mode. You can enable it by commenting out the following line (i.e. CONFIG_NILFS_DEBUG=y) in nilfs2-module/fs/Makefile before compiling:

ifndef CONFIG_NILFS
EXTERNAL_BUILD=y
CONFIG_NILFS=m
# Uncomment below to do debug build. CONFIG_NILFS_DEBUG=y
# Uncomment below to enable bmap validity check. #CONFIG_NILFS_BMAP_DEBUG=y
endif

By the way, I'm planning to release nilfs-2.0.17 tomorrow in order to solve file system corruption problems which infrequently happen and were reported on this list.

The bugfix was already merged in the mainline and also sent to -stable trees for 3.6.30 and 3.6.31, but not yet done.

Your problem looks hardware problem to me, but I think the new version is worth a try.

Cheers,
Ryusuke Konishi
_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

As per my previous question...

I'm running SLES 11, 2.6.27.19-5-default with NILFS2 nilfs-2.0.16. I have a 1.5Tb NILFS2 partition which I am setting up with the intention of using Robocopy from various PCs via samba. The robocopy scripts run nightly and a checkpoint is taken once night. A script stops samba, unmounts the previous weeks checkpoint, deletes the checkpoint, creates a new one and then mounts it and restarts samba. This should mean that at any time the user can go back to 'snapshot_{DAY}' to get their files back.

So far so good.

I'm really confused by the space utilisation I'm seeing and what cleanerd actually does.

The drive is now up to 98% full but should only have about 500Gb on it. I have removed the few checkpoint I made so no all there is are the snapshots. Turning the option in nilfs_cleanerd.conf to debug shows.

Oct  2 16:37:44 cpli6008 nilfs_cleanerd[3611]: start
Oct  2 16:37:44 cpli6008 nilfs_cleanerd[3611]: ncleansegs = 13850
Oct  2 16:38:50 cpli6008 nilfs_cleanerd[3611]: 2 segments selected to be

cleaned

I just used the default options on the mkfs and haven't changed any setting (other than debug) in nilfs_cleanerd.conf.

I'm guessing that my cpio -pdumv from a different machine into the nilfs file system has resulted masses of snapshots and these are taking the space.

Is there a way to clean up or change the preferences to create fewer snapshots?

Many thanks
David Barham
Siemens PLM Software

_______________________________________________
users mailing list
[email protected]
https://www.nilfs.org/mailman/listinfo/users

There's a parameter on nilfs_cleanerd.conf called protection_period. It means checkpoints are kept intact at least that many seconds. You may want to lower that. In addition, you can lower cleaning_interval to make nilfs_cleanerd act more often, and raise nsegments_per_clean to perform a more aggressive (and in my experience, CPU intensive) cleaning.

Unless I'm doing insanely heavy writes or I have very little space left, I find the defaults to work just fine, but yes, some kind of autotuning feature in nilfs_cleanerd would be great.

Hi folks,

I've dug into the iozone code and found it littered with sync(2) calls to flush the buffers. This is what creates the checkpoints. If you take a big enough disc it wouldn't matter otherwise you'll get into dynamic behaviour IOzone is not really build for.

With regards,
Reinoud