Skip to contentSkip to conversation

2.6.20-rc1: CIFS cheers, NFS4 jeers

DISCUSSION Florin Iucha 3 participants 8 messages Feb 26, 2007 – Mar 2, 2007

lore
Share & tools
lore
#0 lore raw

Hello, it's me and my 70 GB of photos again.

I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.

Configuration:

Server: PIII/1GHz, 512 MB RAM, Debian testing,
distro kernel 2.6.18-3-vserver-686, Intel E1000 NIC, filesystem 170 GB ext3 with default mkfs values on a SATA disk

Client: AMD x2 4200+, 2 GB RAM, Debian testing/unstable
kernel 2.6.20-rc1, Marvell SKGE onboard,
filesystem 120 GB ext3 with default mkfs values on a SATA disk

After the writing stalls, I have echoed 't' into /proc/sysrq-trigger and got a trace, which is at http://iucha.net/20-rc1/after.1. There was no oops before the trace request; the 'before' dmesg is at
http://iucha.net/20-rc1/before.1 .

Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.

florin

Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
#1 lore raw

Correction 2.6.21-rc1 Was: 2.6.20-rc1: CIFS cheers, NFS4 jeers

Quoted 4 lines · Florin

After the writing stalls, I have echoed 't' into /proc/sysrq-trigger and got a trace, which is at http://iucha.net/20-rc1/after.1. There was no oops before the trace request; the 'before' dmesg is at
http://iucha.net/20-rc1/before.1 .

The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.

I have corrected the links:

http://iucha.net/21-rc1/before.1
http://iucha.net/21-rc1/after.1
http://iucha.net/21-rc1/config-2.6.21-rc1

florin

Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
#2 lore raw
In reply to Florin Iucha
Quoted 7 lines · Florin Iucha

Hello, it's me and my 70 GB of photos again.

I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.

Configuration:

Quoted 8 lines

Server: PIII/1GHz, 512 MB RAM, Debian testing,
distro kernel 2.6.18-3-vserver-686, Intel E1000 NIC, filesystem 170 GB ext3 with default mkfs values on a SATA disk

Client: AMD x2 4200+, 2 GB RAM, Debian testing/unstable
kernel 2.6.20-rc1, Marvell SKGE onboard,
filesystem 120 GB ext3 with default mkfs values on a SATA disk

Neil has been diddling NFS, I did some light testing with 2.6.20-git14 with 190GB of mp3 and mpg files (library of congress folk music) without hangs. Just "did it work" tests, copy 20-30GB to server, do md5 on the data pulled back from the server.

Didn't hang, performance testing later.

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot
#3 lore raw

Re: 2.6.21-rc1: CIFS cheers, NFS4 jeers

Quoted 6 lines · Bill Davidsen
Quoted 4 lines · Florin Iucha

Hello, it's me and my 70 GB of photos again.

I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.

Quoted 6 lines

Neil has been diddling NFS, I did some light testing with 2.6.20-git14 with 190GB of mp3 and mpg files (library of congress folk music) without hangs. Just "did it work" tests, copy 20-30GB to server, do md5 on the data pulled back from the server.

Didn't hang, performance testing later.

2.6.20-rcX used to copy all files then hang on certain operations that I think used the VFS. 2.6.21-rc1 stalls the NFS transfer itself after several GB. The data was never corrupted.

Have you tried copying _ALL_ 190 GB to the server?

florin

Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
#4 lore raw

Re: 2.6.21-rc1: CIFS cheers, NFS4 jeers

Quoted 13 lines · Florin Iucha
Quoted 11 lines · Bill Davidsen
Quoted 5 lines · Florin Iucha

Hello, it's me and my 70 GB of photos again.

I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.

Neil has been diddling NFS, I did some light testing with 2.6.20-git14 with 190GB of mp3 and mpg files (library of congress folk music) without hangs. Just "did it work" tests, copy 20-30GB to server, do md5 on the data pulled back from the server.

Quoted 8 lines
Quoted 2 lines

Didn't hang, performance testing later.

2.6.20-rcX used to copy all files then hang on certain operations that I think used the VFS. 2.6.21-rc1 stalls the NFS transfer itself after several GB. The data was never corrupted.

Have you tried copying _ALL_ 190 GB to the server?

No, but as noted I was doing 20-30GB, so if "several" is a small number I'm not seeing that behavior. I'm using a Gbit connection if that is not the same as your setup. I have additional testing queued for time available this week.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#5 lore raw
In reply to Florin Iucha
Quoted 15 lines · On Mon, 26 Feb 2007 00:45:00 -0600 [email protected] (Florin Iucha) wrote:

Hello, it's me and my 70 GB of photos again.

I have tested both CIFS and NFSv4 clients in kernel 2.6.20-rc1 . CIFS passed with flying colors and NFSv4 stalled after 7 GB.

Configuration:

Server: PIII/1GHz, 512 MB RAM, Debian testing,
distro kernel 2.6.18-3-vserver-686, Intel E1000 NIC, filesystem 170 GB ext3 with default mkfs values on a SATA disk

Client: AMD x2 4200+, 2 GB RAM, Debian testing/unstable
kernel 2.6.20-rc1, Marvell SKGE onboard,
filesystem 120 GB ext3 with default mkfs values on a SATA disk

Quoted 8 lines

After the writing stalls, I have echoed 't' into /proc/sysrq-trigger and got a trace, which is at http://iucha.net/20-rc1/after.1. There was no oops before the trace request; the 'before' dmesg is at
http://iucha.net/20-rc1/before.1 .

Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.

and

Quoted 11 lines

The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.

I have corrected the links:

   http://iucha.net/21-rc1/before.1
   http://iucha.net/21-rc1/after.1
   http://iucha.net/21-rc1/config-2.6.21-rc1

The relevant part is:

[ 1215.657827] cp            D 000000f86f105704     0  2859   2843                     (NOTLB)
[ 1215.657833]  ffff81007343faa8 0000000000000082 0000000000000000 ffff81007343fb58
[ 1215.657837]  0000000000000002 ffff81007343faa8 0000000000000008 ffff81007e578ee0
[ 1215.657842]  ffff810002f4a080 0000000000002150 ffff81007e5790b8 000000017343fb50
[ 1215.657847] Call Trace:
[ 1215.657852]  [<ffffffff80160f4c>] io_schedule+0x28/0x34
[ 1215.657856]  [<ffffffff80128517>] sync_page+0x41/0x45
[ 1215.657859]  [<ffffffff80161294>] __wait_on_bit+0x45/0x77
[ 1215.657862]  [<ffffffff801284d6>] sync_page+0x0/0x45
[ 1215.657867]  [<ffffffff8013370c>] wait_on_page_bit+0x6e/0x75
[ 1215.657870]  [<ffffffff801948ca>] wake_bit_function+0x0/0x2a
[ 1215.657874]  [<ffffffff801454a2>] pagevec_lookup_tag+0x22/0x2b
[ 1215.657878]  [<ffffffff80147317>] wait_on_page_writeback_range+0x6e/0x142
[ 1215.657885]  [<ffffffff801a4a79>] filemap_fdatawait+0x20/0x22
[ 1215.657889]  [<ffffffff801a4bfe>] filemap_write_and_wait+0x29/0x38
[ 1215.657894]  [<ffffffff8023019a>] nfs_setattr+0xa0/0x11a
[ 1215.657897]  [<ffffffff8010e30e>] link_path_walk+0xe8/0xfc
[ 1215.657902]  [<ffffffff80194892>] autoremove_wake_function+0x0/0x38
[ 1215.657907]  [<ffffffff80112377>] poison_obj+0x27/0x32
[ 1215.657910]  [<ffffffff8010db3a>] current_fs_time+0x3f/0x41
[ 1215.657913]  [<ffffffff80122f93>] __user_walk_fd+0x53/0x62
[ 1215.657918]  [<ffffffff8012bc1b>] notify_change+0x129/0x238
[ 1215.657923]  [<ffffffff801c5ccb>] do_utimes+0xfc/0x126
[ 1215.657928]  [<ffffffff801077fc>] _raw_spin_lock+0xf3/0xf9
[ 1215.657933]  [<ffffffff801c5d3a>] sys_futimesat+0x45/0x56
[ 1215.657937]  [<ffffffff80150d3e>] sys_utimes+0x14/0x16
[ 1215.657941]  [<ffffffff8015ac5e>] system_call+0x7e/0x83

seems that we've simply lost an IO completion.

Was 2.6.19 OK?

#6 lore raw
Quoted 3 lines · Andrew Morton
Quoted 1 line · On Mon, 26 Feb 2007 00:45:00 -0600 [email protected] (Florin Iucha) wrote:

Hello, it's me and my 70 GB of photos again.

[snip]

Quoted 47 lines
Quoted 3 lines

Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.

and

Quoted 10 lines

The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.

I have corrected the links:

   http://iucha.net/21-rc1/before.1
   http://iucha.net/21-rc1/after.1
   http://iucha.net/21-rc1/config-2.6.21-rc1

The relevant part is:

[ 1215.657827] cp            D 000000f86f105704     0  2859   2843                     (NOTLB)
[ 1215.657833]  ffff81007343faa8 0000000000000082 0000000000000000 ffff81007343fb58
[ 1215.657837]  0000000000000002 ffff81007343faa8 0000000000000008 ffff81007e578ee0
[ 1215.657842]  ffff810002f4a080 0000000000002150 ffff81007e5790b8 000000017343fb50
[ 1215.657847] Call Trace:
[ 1215.657852]  [<ffffffff80160f4c>] io_schedule+0x28/0x34
[ 1215.657856]  [<ffffffff80128517>] sync_page+0x41/0x45
[ 1215.657859]  [<ffffffff80161294>] __wait_on_bit+0x45/0x77
[ 1215.657862]  [<ffffffff801284d6>] sync_page+0x0/0x45
[ 1215.657867]  [<ffffffff8013370c>] wait_on_page_bit+0x6e/0x75
[ 1215.657870]  [<ffffffff801948ca>] wake_bit_function+0x0/0x2a
[ 1215.657874]  [<ffffffff801454a2>] pagevec_lookup_tag+0x22/0x2b
[ 1215.657878]  [<ffffffff80147317>] wait_on_page_writeback_range+0x6e/0x142
[ 1215.657885]  [<ffffffff801a4a79>] filemap_fdatawait+0x20/0x22
[ 1215.657889]  [<ffffffff801a4bfe>] filemap_write_and_wait+0x29/0x38
[ 1215.657894]  [<ffffffff8023019a>] nfs_setattr+0xa0/0x11a
[ 1215.657897]  [<ffffffff8010e30e>] link_path_walk+0xe8/0xfc
[ 1215.657902]  [<ffffffff80194892>] autoremove_wake_function+0x0/0x38
[ 1215.657907]  [<ffffffff80112377>] poison_obj+0x27/0x32
[ 1215.657910]  [<ffffffff8010db3a>] current_fs_time+0x3f/0x41
[ 1215.657913]  [<ffffffff80122f93>] __user_walk_fd+0x53/0x62
[ 1215.657918]  [<ffffffff8012bc1b>] notify_change+0x129/0x238
[ 1215.657923]  [<ffffffff801c5ccb>] do_utimes+0xfc/0x126
[ 1215.657928]  [<ffffffff801077fc>] _raw_spin_lock+0xf3/0xf9
[ 1215.657933]  [<ffffffff801c5d3a>] sys_futimesat+0x45/0x56
[ 1215.657937]  [<ffffffff80150d3e>] sys_utimes+0x14/0x16
[ 1215.657941]  [<ffffffff8015ac5e>] system_call+0x7e/0x83
Quoted 3 lines

seems that we've simply lost an IO completion.

Was 2.6.19 OK?

I just tested, 2.6.19 is OK! Kernel log output after the cp and sync completed are at

http://iucha.net/19/before
http://iucha.net/19/after (after echo t > /proc/sysrq-trigger)

When I get a chance I will try again, and report if it fails. But so far it seems fine: df and lsof work as expected.

Thanks,
florin

Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163
#7 lore raw
Quoted 52 lines · Florin Iucha
Quoted 3 lines · Andrew Morton
Quoted 1 line · On Mon, 26 Feb 2007 00:45:00 -0600 [email protected] (Florin Iucha) wrote:

Hello, it's me and my 70 GB of photos again.

[snip]

Quoted 47 lines
Quoted 3 lines

Running 'top', one core is idle and the other is 99% waiting, while the 'cp' program is in 'D' state. Also, after NFSv4 stalls, invokations of 'lsof' stall as well. I can 'ssh' into the box without problems.

and

Quoted 10 lines

The kernel on the client is 2.6.21-rc1 (but it echoes problems I reported in December with 2.6.20 series as well) as can be seen from the kernel logs.

I have corrected the links:

   http://iucha.net/21-rc1/before.1
   http://iucha.net/21-rc1/after.1
   http://iucha.net/21-rc1/config-2.6.21-rc1

The relevant part is:

[ 1215.657827] cp            D 000000f86f105704     0  2859   2843                     (NOTLB)
[ 1215.657833]  ffff81007343faa8 0000000000000082 0000000000000000 ffff81007343fb58
[ 1215.657837]  0000000000000002 ffff81007343faa8 0000000000000008 ffff81007e578ee0
[ 1215.657842]  ffff810002f4a080 0000000000002150 ffff81007e5790b8 000000017343fb50
[ 1215.657847] Call Trace:
[ 1215.657852]  [<ffffffff80160f4c>] io_schedule+0x28/0x34
[ 1215.657856]  [<ffffffff80128517>] sync_page+0x41/0x45
[ 1215.657859]  [<ffffffff80161294>] __wait_on_bit+0x45/0x77
[ 1215.657862]  [<ffffffff801284d6>] sync_page+0x0/0x45
[ 1215.657867]  [<ffffffff8013370c>] wait_on_page_bit+0x6e/0x75
[ 1215.657870]  [<ffffffff801948ca>] wake_bit_function+0x0/0x2a
[ 1215.657874]  [<ffffffff801454a2>] pagevec_lookup_tag+0x22/0x2b
[ 1215.657878]  [<ffffffff80147317>] wait_on_page_writeback_range+0x6e/0x142
[ 1215.657885]  [<ffffffff801a4a79>] filemap_fdatawait+0x20/0x22
[ 1215.657889]  [<ffffffff801a4bfe>] filemap_write_and_wait+0x29/0x38
[ 1215.657894]  [<ffffffff8023019a>] nfs_setattr+0xa0/0x11a
[ 1215.657897]  [<ffffffff8010e30e>] link_path_walk+0xe8/0xfc
[ 1215.657902]  [<ffffffff80194892>] autoremove_wake_function+0x0/0x38
[ 1215.657907]  [<ffffffff80112377>] poison_obj+0x27/0x32
[ 1215.657910]  [<ffffffff8010db3a>] current_fs_time+0x3f/0x41
[ 1215.657913]  [<ffffffff80122f93>] __user_walk_fd+0x53/0x62
[ 1215.657918]  [<ffffffff8012bc1b>] notify_change+0x129/0x238
[ 1215.657923]  [<ffffffff801c5ccb>] do_utimes+0xfc/0x126
[ 1215.657928]  [<ffffffff801077fc>] _raw_spin_lock+0xf3/0xf9
[ 1215.657933]  [<ffffffff801c5d3a>] sys_futimesat+0x45/0x56
[ 1215.657937]  [<ffffffff80150d3e>] sys_utimes+0x14/0x16
[ 1215.657941]  [<ffffffff8015ac5e>] system_call+0x7e/0x83
Quoted 6 lines
Quoted 3 lines

seems that we've simply lost an IO completion.

Was 2.6.19 OK?

I just tested, 2.6.19 is OK! Kernel log output after the cp and sync completed are at

Second test on 2.6.19 and passes.

florin

Signature · 2 lines
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163