2.6.20-rc1: CIFS cheers, NFS4 jeers
Hello, it's me and my 70 GB of photos again.
I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.
Configuration:
Server: PIII/1GHz, 512 MB RAM, Debian testing,
distro kernel 2.6.18-3-vserver-686, Intel E1000 NIC, filesystem 170 GB ext3 with default mkfs values on a SATA disk
Client: AMD x2 4200+, 2 GB RAM, Debian testing/unstable
kernel 2.6.20-rc1, Marvell SKGE onboard,
filesystem 120 GB ext3 with default mkfs values on a SATA disk
After the writing stalls, I have echoed 't' into /proc/sysrq-trigger and got a trace, which is at http://iucha.net/20-rc1/after.1. There was no oops before the trace request; the 'before' dmesg is at
http://iucha.net/20-rc1/before.1 .
Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.
florin
Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
http://geekz.co.uk/schneierfacts/fact/163Correction 2.6.21-rc1 Was: 2.6.20-rc1: CIFS cheers, NFS4 jeers
Quoted 4 lines · Florin
After the writing stalls, I have echoed 't' into
/proc/sysrq-triggerand got a trace, which is at http://iucha.net/20-rc1/after.1. There was no oops before the trace request; the 'before' dmesg is at
http://iucha.net/20-rc1/before.1 .
The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.
I have corrected the links:
http://iucha.net/21-rc1/before.1
http://iucha.net/21-rc1/after.1
http://iucha.net/21-rc1/config-2.6.21-rc1
florin
Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
http://geekz.co.uk/schneierfacts/fact/163Quoted 7 lines · Florin Iucha
Hello, it's me and my 70 GB of photos again.
I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.
Configuration:
Quoted 8 lines
Server: PIII/1GHz, 512 MB RAM, Debian testing,
distro kernel 2.6.18-3-vserver-686, Intel E1000 NIC, filesystem 170 GB ext3 with default mkfs values on a SATA diskClient: AMD x2 4200+, 2 GB RAM, Debian testing/unstable
kernel 2.6.20-rc1, Marvell SKGE onboard,
filesystem 120 GB ext3 with default mkfs values on a SATA disk
Neil has been diddling NFS, I did some light testing with 2.6.20-git14 with 190GB of mp3 and mpg files (library of congress folk music) without hangs. Just "did it work" tests, copy 20-30GB to server, do md5 on the data pulled back from the server.
Didn't hang, performance testing later.
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot
Re: 2.6.21-rc1: CIFS cheers, NFS4 jeers
Quoted 6 lines · Bill Davidsen
Quoted 4 lines · Florin Iucha
Hello, it's me and my 70 GB of photos again.
I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.
Quoted 6 lines
Neil has been diddling NFS, I did some light testing with 2.6.20-git14 with 190GB of mp3 and mpg files (library of congress folk music) without hangs. Just "did it work" tests, copy 20-30GB to server, do md5 on the data pulled back from the server.
Didn't hang, performance testing later.
2.6.20-rcX used to copy all files then hang on certain operations that I think used the VFS. 2.6.21-rc1 stalls the NFS transfer itself after several GB. The data was never corrupted.
Have you tried copying _ALL_ 190 GB to the server?
florin
Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
http://geekz.co.uk/schneierfacts/fact/163Re: 2.6.21-rc1: CIFS cheers, NFS4 jeers
Quoted 13 lines · Florin Iucha
Quoted 11 lines · Bill Davidsen
Quoted 5 lines · Florin Iucha
Hello, it's me and my 70 GB of photos again.
I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.
Neil has been diddling NFS, I did some light testing with 2.6.20-git14 with 190GB of mp3 and mpg files (library of congress folk music) without hangs. Just "did it work" tests, copy 20-30GB to server, do md5 on the data pulled back from the server.
Quoted 8 lines
Quoted 2 lines
Didn't hang, performance testing later.
2.6.20-rcX used to copy all files then hang on certain operations that I think used the VFS. 2.6.21-rc1 stalls the NFS transfer itself after several GB. The data was never corrupted.
Have you tried copying _ALL_ 190 GB to the server?
No, but as noted I was doing 20-30GB, so if "several" is a small number I'm not seeing that behavior. I'm using a Gbit connection if that is not the same as your setup. I have additional testing queued for time available this week.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Quoted 15 lines · On Mon, 26 Feb 2007 00:45:00 -0600 [email protected] (Florin Iucha) wrote:
Hello, it's me and my 70 GB of photos again.
I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.
Configuration:
Server: PIII/1GHz, 512 MB RAM, Debian testing,
distro kernel 2.6.18-3-vserver-686, Intel E1000 NIC, filesystem 170 GB ext3 with default mkfs values on a SATA diskClient: AMD x2 4200+, 2 GB RAM, Debian testing/unstable
kernel 2.6.20-rc1, Marvell SKGE onboard,
filesystem 120 GB ext3 with default mkfs values on a SATA disk
Quoted 8 lines
After the writing stalls, I have echoed 't' into
/proc/sysrq-triggerand got a trace, which is at http://iucha.net/20-rc1/after.1. There was no oops before the trace request; the 'before' dmesg is at
http://iucha.net/20-rc1/before.1 .Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.
and
Quoted 11 lines
The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.
I have corrected the links:
http://iucha.net/21-rc1/before.1 http://iucha.net/21-rc1/after.1 http://iucha.net/21-rc1/config-2.6.21-rc1
The relevant part is:
[ 1215.657827] cp D 000000f86f105704 0 2859 2843 (NOTLB) [ 1215.657833] ffff81007343faa8 0000000000000082 0000000000000000 ffff81007343fb58 [ 1215.657837] 0000000000000002 ffff81007343faa8 0000000000000008 ffff81007e578ee0 [ 1215.657842] ffff810002f4a080 0000000000002150 ffff81007e5790b8 000000017343fb50 [ 1215.657847] Call Trace: [ 1215.657852] [<ffffffff80160f4c>] io_schedule+0x28/0x34 [ 1215.657856] [<ffffffff80128517>] sync_page+0x41/0x45 [ 1215.657859] [<ffffffff80161294>] __wait_on_bit+0x45/0x77 [ 1215.657862] [<ffffffff801284d6>] sync_page+0x0/0x45 [ 1215.657867] [<ffffffff8013370c>] wait_on_page_bit+0x6e/0x75 [ 1215.657870] [<ffffffff801948ca>] wake_bit_function+0x0/0x2a [ 1215.657874] [<ffffffff801454a2>] pagevec_lookup_tag+0x22/0x2b [ 1215.657878] [<ffffffff80147317>] wait_on_page_writeback_range+0x6e/0x142 [ 1215.657885] [<ffffffff801a4a79>] filemap_fdatawait+0x20/0x22 [ 1215.657889] [<ffffffff801a4bfe>] filemap_write_and_wait+0x29/0x38 [ 1215.657894] [<ffffffff8023019a>] nfs_setattr+0xa0/0x11a [ 1215.657897] [<ffffffff8010e30e>] link_path_walk+0xe8/0xfc [ 1215.657902] [<ffffffff80194892>] autoremove_wake_function+0x0/0x38 [ 1215.657907] [<ffffffff80112377>] poison_obj+0x27/0x32 [ 1215.657910] [<ffffffff8010db3a>] current_fs_time+0x3f/0x41 [ 1215.657913] [<ffffffff80122f93>] __user_walk_fd+0x53/0x62 [ 1215.657918] [<ffffffff8012bc1b>] notify_change+0x129/0x238 [ 1215.657923] [<ffffffff801c5ccb>] do_utimes+0xfc/0x126 [ 1215.657928] [<ffffffff801077fc>] _raw_spin_lock+0xf3/0xf9 [ 1215.657933] [<ffffffff801c5d3a>] sys_futimesat+0x45/0x56 [ 1215.657937] [<ffffffff80150d3e>] sys_utimes+0x14/0x16 [ 1215.657941] [<ffffffff8015ac5e>] system_call+0x7e/0x83
seems that we've simply lost an IO completion.
Was 2.6.19 OK?
Quoted 3 lines · Andrew Morton
Quoted 1 line · On Mon, 26 Feb 2007 00:45:00 -0600 [email protected] (Florin Iucha) wrote:
Hello, it's me and my 70 GB of photos again.
[snip]
Quoted 47 lines
Quoted 3 lines
Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.
and
Quoted 10 lines
The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.
I have corrected the links:
http://iucha.net/21-rc1/before.1 http://iucha.net/21-rc1/after.1 http://iucha.net/21-rc1/config-2.6.21-rc1The relevant part is:
[ 1215.657827] cp D 000000f86f105704 0 2859 2843 (NOTLB) [ 1215.657833] ffff81007343faa8 0000000000000082 0000000000000000 ffff81007343fb58 [ 1215.657837] 0000000000000002 ffff81007343faa8 0000000000000008 ffff81007e578ee0 [ 1215.657842] ffff810002f4a080 0000000000002150 ffff81007e5790b8 000000017343fb50 [ 1215.657847] Call Trace: [ 1215.657852] [<ffffffff80160f4c>] io_schedule+0x28/0x34 [ 1215.657856] [<ffffffff80128517>] sync_page+0x41/0x45 [ 1215.657859] [<ffffffff80161294>] __wait_on_bit+0x45/0x77 [ 1215.657862] [<ffffffff801284d6>] sync_page+0x0/0x45 [ 1215.657867] [<ffffffff8013370c>] wait_on_page_bit+0x6e/0x75 [ 1215.657870] [<ffffffff801948ca>] wake_bit_function+0x0/0x2a [ 1215.657874] [<ffffffff801454a2>] pagevec_lookup_tag+0x22/0x2b [ 1215.657878] [<ffffffff80147317>] wait_on_page_writeback_range+0x6e/0x142 [ 1215.657885] [<ffffffff801a4a79>] filemap_fdatawait+0x20/0x22 [ 1215.657889] [<ffffffff801a4bfe>] filemap_write_and_wait+0x29/0x38 [ 1215.657894] [<ffffffff8023019a>] nfs_setattr+0xa0/0x11a [ 1215.657897] [<ffffffff8010e30e>] link_path_walk+0xe8/0xfc [ 1215.657902] [<ffffffff80194892>] autoremove_wake_function+0x0/0x38 [ 1215.657907] [<ffffffff80112377>] poison_obj+0x27/0x32 [ 1215.657910] [<ffffffff8010db3a>] current_fs_time+0x3f/0x41 [ 1215.657913] [<ffffffff80122f93>] __user_walk_fd+0x53/0x62 [ 1215.657918] [<ffffffff8012bc1b>] notify_change+0x129/0x238 [ 1215.657923] [<ffffffff801c5ccb>] do_utimes+0xfc/0x126 [ 1215.657928] [<ffffffff801077fc>] _raw_spin_lock+0xf3/0xf9 [ 1215.657933] [<ffffffff801c5d3a>] sys_futimesat+0x45/0x56 [ 1215.657937] [<ffffffff80150d3e>] sys_utimes+0x14/0x16 [ 1215.657941] [<ffffffff8015ac5e>] system_call+0x7e/0x83
Quoted 3 lines
seems that we've simply lost an IO completion.
Was 2.6.19 OK?
I just tested, 2.6.19 is OK! Kernel log output after the cp and sync completed are at
http://iucha.net/19/before
http://iucha.net/19/after (after echo t > /proc/sysrq-trigger)
When I get a chance I will try again, and report if it fails. But so far it seems fine: df and lsof work as expected.
Thanks,
florin
Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
http://geekz.co.uk/schneierfacts/fact/163Quoted 52 lines · Florin Iucha
Quoted 3 lines · Andrew Morton
Quoted 1 line · On Mon, 26 Feb 2007 00:45:00 -0600 [email protected] (Florin Iucha) wrote:
Hello, it's me and my 70 GB of photos again.
[snip]
Quoted 47 lines
Quoted 3 lines
Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.
and
Quoted 10 lines
The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.
I have corrected the links:
http://iucha.net/21-rc1/before.1 http://iucha.net/21-rc1/after.1 http://iucha.net/21-rc1/config-2.6.21-rc1The relevant part is:
[ 1215.657827] cp D 000000f86f105704 0 2859 2843 (NOTLB) [ 1215.657833] ffff81007343faa8 0000000000000082 0000000000000000 ffff81007343fb58 [ 1215.657837] 0000000000000002 ffff81007343faa8 0000000000000008 ffff81007e578ee0 [ 1215.657842] ffff810002f4a080 0000000000002150 ffff81007e5790b8 000000017343fb50 [ 1215.657847] Call Trace: [ 1215.657852] [<ffffffff80160f4c>] io_schedule+0x28/0x34 [ 1215.657856] [<ffffffff80128517>] sync_page+0x41/0x45 [ 1215.657859] [<ffffffff80161294>] __wait_on_bit+0x45/0x77 [ 1215.657862] [<ffffffff801284d6>] sync_page+0x0/0x45 [ 1215.657867] [<ffffffff8013370c>] wait_on_page_bit+0x6e/0x75 [ 1215.657870] [<ffffffff801948ca>] wake_bit_function+0x0/0x2a [ 1215.657874] [<ffffffff801454a2>] pagevec_lookup_tag+0x22/0x2b [ 1215.657878] [<ffffffff80147317>] wait_on_page_writeback_range+0x6e/0x142 [ 1215.657885] [<ffffffff801a4a79>] filemap_fdatawait+0x20/0x22 [ 1215.657889] [<ffffffff801a4bfe>] filemap_write_and_wait+0x29/0x38 [ 1215.657894] [<ffffffff8023019a>] nfs_setattr+0xa0/0x11a [ 1215.657897] [<ffffffff8010e30e>] link_path_walk+0xe8/0xfc [ 1215.657902] [<ffffffff80194892>] autoremove_wake_function+0x0/0x38 [ 1215.657907] [<ffffffff80112377>] poison_obj+0x27/0x32 [ 1215.657910] [<ffffffff8010db3a>] current_fs_time+0x3f/0x41 [ 1215.657913] [<ffffffff80122f93>] __user_walk_fd+0x53/0x62 [ 1215.657918] [<ffffffff8012bc1b>] notify_change+0x129/0x238 [ 1215.657923] [<ffffffff801c5ccb>] do_utimes+0xfc/0x126 [ 1215.657928] [<ffffffff801077fc>] _raw_spin_lock+0xf3/0xf9 [ 1215.657933] [<ffffffff801c5d3a>] sys_futimesat+0x45/0x56 [ 1215.657937] [<ffffffff80150d3e>] sys_utimes+0x14/0x16 [ 1215.657941] [<ffffffff8015ac5e>] system_call+0x7e/0x83
Quoted 6 lines
Quoted 3 lines
seems that we've simply lost an IO completion.
Was 2.6.19 OK?
I just tested, 2.6.19 is OK! Kernel log output after the cp and sync completed are at
Second test on 2.6.19 and passes.
florin
Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
http://geekz.co.uk/schneierfacts/fact/163