IO scheduling & filesystem v a few processes writing a lot
Hi,
I've got a backup system that I'm trying to eek some more performance
out of - and I don't understand the right way to get the kernel to do disc writes efficiently - and hence would like some advice.
I was using rsync, but the problem with rsync is that I have a back up server then filled with lots and lots of small files
- I want larger files for spooling to tape.
(Other suggestions welcome)
So I'm trying switching to streaming gzip'd tars from each client to backup to the server. I have one server that opens connections to each of the clients and sucks the data using netcat (now netcat6 in ipv4 mode) and writes it to
disc, one file per client. Now the downside here
relative to rsync is that it is going to transfer and write a lot more data.
Now the clients are on 100Mb/s, and the server on GigE,
the clients sometime have to think while they gzip their data, so I'd like to suck data from multiple clients at once. So I run multiple of these netcat's in parallel - currently about 9.
I've benchmarked write performance on the filesystem at 60-70MB/s for a single write process (as shown with iostat) for a simple dd if=/dev/zero of=abigfile bs=1024k
My problem is that with the parallel writes iostat is showing I'm actually getting ~3MB/s write bandwidth - that stinks!
The machine is a dual xeon with 1GB of RAM, an intel GigE card and a 2.6.11 kernel, a 3ware-9000 series pci-x controller with a 1.5TB RAID5 partition running Reiser3. Reiser3 is used because I couldn't get ext3 stable on a filesystem of this size (-64ZByte free shown in df), and xfs didn't seem stable on recovering from an arbitrarily placed reset. The 3ware has write caching (with battery backup). (Note this is the 9000 series SATA ones, not the older 7000/8000 that really did suck when writing). The CFQ scheduler is being used and I've turned up nr_requests to 1024.
So as I see it I have to persuade all appropriate parts to buffer sensibly and try and generate as large I/O requests as possible; I've told netcat6 (on the read side) to use 128k buffers:
nc6 -4 --recv-only --buffer-size=131072 --idle-timeout=3600
Iostat is showing the world is not happy:
avg-cpu: %user %nice %system %iowait %idle
0.15 0.00 2.65 97.20 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
sda 0.00 249.95 0.10 193.62 0.80 3548.55 0.40 1774.28
avgrq-sz avgqu-sz await svctm %util
18.32 1290.24 8225.93 5.15 99.73
avg-cpu: %user %nice %system %iowait %idle
0.15 0.00 3.05 95.95 0.85
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
sda 0.00 260.96 0.00 217.62 0.00 4342.74 0.00 2171.37
avgrq-sz avgqu-sz await svctm %util
19.96 1630.13 6951.26 4.60 100.12
(I could do with a pointer to something explaining all the fields here!)
But as I read this, the machine is stuck waiting on I/O (OK), but only managing to write ~2MB/s - and seems to have done that 2MB in about 200 writes (am I reading that correctly?) - which is rather on the small side.
So I can understand it going slowly if it is interleaving all the requests at tiny 10KB writes all over the disc - how do I persuade it I'd like it to buffer more and get larger writes - or how do I understand why it isn't?
At the moment it is feeling like I'm going to have to write something that has a single process doing the writes and accumulates the data from all the clients itself so that there is only ever one process writing.
I'm open for all suggestions.
Dave
--
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _________________________|_____ http://www.treblig.org |_______/
Dr. David Alan Gilbert wrote (ao):
Quoted 8 lines
I've got a backup system that I'm trying to eek some more performance
out of - and I don't understand the right way to get the kernel to do disc writes efficiently - and hence would like some advice.I was using rsync, but the problem with rsync is that I have a back up server then filled with lots and lots of small files
- I want larger files for spooling to tape.
(Other suggestions welcome)
Can't you just tar the small files from the backupserver to tape? (or, what is the problem with that?).
Quoted 7 lines
So I'm trying switching to streaming gzip'd tars from each client to backup to the server. I have one server that opens connections to each of the clients and sucks the data using netcat (now netcat6 in ipv4 mode) and writes it to disc, one file per client. Now the downside here relative to rsync is that it is going to transfer and write a lot more data.
You also do incremental backups?
Quoted 11 lines
Now the clients are on 100Mb/s, and the server on GigE,
the clients sometime have to think while they gzip their data, so I'd like to suck data from multiple clients at once. So I run multiple of these netcat's in parallel - currently about 9.I've benchmarked write performance on the filesystem at 60-70MB/s for a single write process (as shown with iostat) for a simple dd if=/dev/zero of=abigfile bs=1024k
My problem is that with the parallel writes iostat is showing I'm actually getting ~3MB/s write bandwidth - that stinks!
How many parallel streams can the system currently handle before the write bandwith gets unacceptable?
Quoted 3 lines
The machine is a dual xeon with 1GB of RAM, an intel GigE card and a 2.6.11 kernel, a 3ware-9000 series pci-x controller with a 1.5TB RAID5 partition running Reiser3.
What mount options? And how many disks?
Quoted 2 lines
Reiser3 is used because I couldn't get ext3 stable on a filesystem of this size (-64ZByte free shown in df),
That is not a sign of instability per se AFAIK.
Quoted 2 lines
and xfs didn't seem stable on recovering from an arbitrarily placed reset. The 3ware has write caching (with battery backup).
How is the cache configured in the bios?
Quoted 1 line
I'm open for all suggestions.
Would it be possible to test software raid to see if that gives different numbers?
Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Quoted 9 lines · * Sander ([email protected]) wrote:
Dr. David Alan Gilbert wrote (ao):
Quoted 4 lines
I was using rsync, but the problem with rsync is that I have a back up server then filled with lots and lots of small files
- I want larger files for spooling to tape.
(Other suggestions welcome)Can't you just tar the small files from the backupserver to tape? (or, what is the problem with that?).
Lots of small files->slow; it is an LTO-2 tape drive that is spec'd at 35MByte/s - it won't get that if I'm feeding it from something seeking all over.
Quoted 3 lines
Quoted 1 line
write a lot more data.
You also do incremental backups?
I could - but they are a pain at restore time.
Quoted 9 lines
Quoted 6 lines
I've benchmarked write performance on the filesystem at 60-70MB/s for a single write process (as shown with iostat) for a simple dd if=/dev/zero of=abigfile bs=1024k
My problem is that with the parallel writes iostat is showing I'm actually getting ~3MB/s write bandwidth - that stinks!
How many parallel streams can the system currently handle before the write bandwith gets unacceptable?
I'll be honest I don't know; this was running with 9 streams; but I know the overall speed of the backup goes up as I increase the parallelism from 5 through 9 - but it still sucks.
Quoted 5 lines
Quoted 3 lines
The machine is a dual xeon with 1GB of RAM, an intel GigE card and a 2.6.11 kernel, a 3ware-9000 series pci-x controller with a 1.5TB RAID5 partition running Reiser3.
What mount options? And how many disks?
7 active discs, raid5; mounted with noatime, nodiratime
Quoted 4 lines
Quoted 2 lines
Reiser3 is used because I couldn't get ext3 stable on a filesystem of this size (-64ZByte free shown in df),
That is not a sign of instability per se AFAIK.
When I fsck it fixes it - this to me is an indication something is wrong with the ondisc data; now it might only be the freespace totals - but the fact that the disc contents are wrong makes me worry - I don't like having to fsck a 1.5TB partition.
Quoted 4 lines
Quoted 2 lines
and xfs didn't seem stable on recovering from an arbitrarily placed reset. The 3ware has write caching (with battery backup).
How is the cache configured in the bios?
Write cache is on in the 3ware bios as is the battery backup.
Quoted 4 lines
Quoted 1 line
I'm open for all suggestions.
Would it be possible to test software raid to see if that gives different numbers?
Erm I guess I could - but the controller does manage 60/70MB/s write as a raw stream, so as far as I can tell if I can persuade the kernel not to chop my writes into silly small chunks things should be good.
Dave
--
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _________________________|_____ http://www.treblig.org |_______/
Dr. David Alan Gilbert wrote (ao):
Quoted 7 lines · * Sander ([email protected]) wrote:
Quoted 6 lines
Dr. David Alan Gilbert wrote (ao):
Quoted 4 lines
I was using rsync, but the problem with rsync is that I have a back up server then filled with lots and lots of small files
- I want larger files for spooling to tape.
(Other suggestions welcome)
Quoted 6 lines
Quoted 2 lines
Can't you just tar the small files from the backupserver to tape? (or, what is the problem with that?).
Lots of small
files->slow; it is an LTO-2 tape drive that is spec'd at 35MByte/s - it won't get that if I'm feeding it from something seeking all over.
ic. Sorry if the question is stupid, but is it bad not to reach 35MB/sec?
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
write a lot more data.
You also do incremental backups?
I could - but they are a pain at restore time.
Well, bare metal restores are rare, and if you need to do one, IMHO one full restore and six incrementals (worst case, and with one full backup a week) are not that painful. Very IMHO of course. You could lessen the pain with incrementals_since_last_full (can't remember the correct term ATM).
If you go with weekly fulls, you can almost have a single system per day streaming a full backup to your backupserver.
Quoted 3 lines
Quoted 1 line
What mount options? And how many disks?
7 active discs, raid5; mounted with noatime, nodiratime
Should perform at least a bit..
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Reiser3 is used because I couldn't get ext3 stable on a filesystem of this size (-64ZByte free shown in df),
That is not a sign of instability per se AFAIK.
When I fsck it fixes it - this to me is an indication something is wrong with the ondisc data; now it might only be the freespace totals
- but the fact that the disc contents are wrong makes me worry - I don't like having to fsck a 1.5TB partition.
Yes, I understand.
Quoted 3 lines
Quoted 1 line
How is the cache configured in the bios?
Write cache is on in the 3ware bios as is the battery backup.
According to the docs it is just 'enable' or 'disable'. I remember raid controllers (Intel?) which also have 'write through' or 'write back'. That was what I was looking for, but no such thing it seems.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
I'm open for all suggestions.
Would it be possible to test software raid to see if that gives different numbers?
Erm I guess I could - but the controller does manage 60/70MB/s write as a raw stream, so as far as I can tell if I can persuade the kernel not to chop my writes into silly small chunks things should be good.
Yes, but if you can lift the performance at some point, it might get acceptable. You could for example also try a raid0 stripe across the 7 disks.
Kind regards, Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Quoted 11 lines · Sander
Dr. David Alan Gilbert wrote (ao):
Quoted 10 lines · * Sander ([email protected]) wrote:
Quoted 8 lines
Dr. David Alan Gilbert wrote (ao):
Quoted 4 lines
I was using rsync, but the problem with rsync is that I have a back up server then filled with lots and lots of small files
- I want larger files for spooling to tape.
(Other suggestions welcome)Can't you just tar the small files from the backupserver to tape? (or, what is the problem with that?).
Quoted 6 lines
Quoted 3 lines
Lots of small
files->slow; it is an LTO-2 tape drive that is spec'd at 35MByte/s - it won't get that if I'm feeding it from something seeking all over.ic. Sorry if the question is stupid, but is it bad not to reach 35MB/sec?
with modern tape drives, when you fall out of streaming mode you are lucky to get 1/10 of the rated drive performance (not to mention the extra wear and tear on the tape and the drive)
the common thing is to do disk->disk->tape backups. use rsync to pull your data from the remote machines to your server, then use tar on the server to make the one image you want to put on the tape (frequently onto a different drive), then record that image to tape.
note that tar is not nessasarily the best format to use for this in the face of tape errors (see backup software companies for details, several years ago I read an interesting document from the bru backup folks that went into details)
David Lang
Signature · 2 lines
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare